Repository navigation
feat(forms): map form fields with a geometric solver when AcroForm widgets are available - #4496
Merged
Merged
Conversation
Extract native interactive PDF widgets from docling-parse parsed pages and materialize them as keyless, format-neutral fillable field values. A small PDF form-field stage runs after OCR and before layout postprocessing — the only point where both the raw layout FORM clusters and the parsed-page widgets are available. It normalizes each widget into a format-neutral field prediction, binds it to a raw FORM cluster (with a page-local fallback region for unmatched widgets), and never mutates the layout prediction or infers labels. Page assembly turns matched FORM clusters into FieldRegionElements and emits fallback regions; reading order materializes them as FieldRegionItem/FieldItem/FieldValueItem in native widget order. Raw AcroForm state (/Ff, /AS, field names) stays on SegmentedPdfPage.widgets; nothing PDF-specific enters DoclingDocument or add_field_value(). The legacy late-pypdfium2 FormItem+GraphData path (form_utils.py) is removed. Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
Document experiment findings on native widget state vs. visual layout extraction colliding for filled form fields, alongside form field model and pipeline updates. Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
Materialize a non-pushbutton /Btn widget as an empty-text FieldValueItem that nests a CHECKBOX_SELECTED/UNSELECTED child, instead of encoding state as a synthesized "checked"/"unchecked" value string. The value text stays empty so the doclang serializer inlines the <checkbox> token into <value>. Keeps /AS as the authoritative state and preserves the raw value in orig. Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
…abel Reconcile the native /Btn widget with the layout CHECKBOX_* cluster it overlaps: match on widget-rect-in-cluster overlap, lift the cluster's option label (mark glyph stripped) onto the nested checkbox child, and promote the cluster out of the plain-text stream so it is no longer a disjoint duplicate. /AS stays authoritative for the selection state, overriding the glyph classifier. Unselected boxes with no overlapping cluster keep the widget-only path. Verified on f1040s1_filled: the checked option serializes as <value class="fillable"><checkbox class="selected"/>4797</value> with the duplicate cluster removed. Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
…ts bbox as prov Drop the mark-glyph sanitation; the checkbox cluster's text is now lifted as-is onto the value, and the matched cluster bbox becomes the value's prov so the box wraps the whole checkbox+label rather than the tiny widget rect.
…ield_item
Introduce an item layer between field region and value: FieldRegionPrediction
now holds FieldItemPrediction items, each a keyless single value (today's flat
shape, unchanged) or a keyed group. A widget with no smaller enclosing container
than a text paragraph is matched to that paragraph -- the smallest enclosing
cluster wins over the page-wide FORM -- and widgets sharing one paragraph
accumulate under a single key (the paragraph text/bbox), with the paragraph
promoted out of the plain-text body. Reading order materializes the key via
add_field_key.
IRS Schedule 1 line 7 ("check here [] and enter amount repaid: 1221.00") now
emits one field_item: key + checkbox-unselected value + fillable "1221.00",
instead of two disjoint value-only fields.
…ld_region An inlined-widget paragraph was pulled out into its own field_region, detaching it from the list (and enclosing form region) it belonged to. Keep the paragraph cluster in the body and attach the keyed item onto its TextElement; reading order now emits a field_item in the paragraph's own place -- inside its list when it is a list item, else under the current parent -- with its text as the key. No separate field_region is created for the inline case. On IRS Sch.1, line 7's field_item now sits inside its list within the form's existing field_region instead of a standalone region ahead of it. Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
A widget of zero height or width is an artifact (Well-Tagged PDF 1.0, 8.9.2.4.13) and a push button holds no field value; both were being emitted as empty fillable values. Skip them before normalization. Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
Pure geometry for the widget-to-key binding: _gap (rectangle edge-gap, direction logic for free), _precedes (no-crossing reading-order guard, line-height row band), and _match_labels (monotonic forward pass, 1:1). Not yet wired into the form-field model -- this is the isolated, fixture-free core so the minimization property is unit-tested on its own. Tests cover the label-steal and crossing cases that the reframing exists to remove, plus 1:1 shared-header exhaustion and the gap cap. Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
Wire the order-preserving binding into the form-field model: after the per-widget triage, run _match_labels over the keyless FORM-matched and unmatched widgets (in widget.index order) against the body label pool (text clusters minus consumed paragraph containers). A bound label becomes the field item's key_text/key_bbox and is promoted out of the body via the existing _drop_clusters machinery, so no new plumbing on the assembly side. Matched checkboxes already carry their option label and stay untouched. Adds the two calibration knobs (gap cap, row band, in line-heights) and a median-line-height text scale; the 15-form harness sets their values. Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
…der gate Two bugs found by inspecting gst494 output against the rendered page: - The keyless guard tested `checkbox_label is None`, but the field defaults to "" -- so no widget ever reached the binding pass and the feature was dead. Gate on falsy instead. - `_precedes` gated the no-crossing guard on left/right order within a fuzzy row band. Binding a right-side label (a trailing "RT") poisoned the frontier so every left-side label in the next row was wrongly ruled "before" it, blocking Contact person / Title / Telephone number (all gap 0). The guard is now purely vertical: block only a strictly-higher row. No left-to-right assumption; 1:1 pairing is handled by per-label consumption alone. On gst494 this takes binding from 0 real keys to 11/12 correct (the one miss is a garbage layout text cluster, not a binding error). Updates the pipeline test: the text field now correctly binds its detached "Full name:" left label. Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
Capture what the fixture inspection taught: the two bugs that made the binding pass dead, the _precedes correction (row-only, no left-order gate), and the accepted table-form column-crossing ceiling (rf-1084s). Add a ponytail note at the binding site pointing at the handoff doc. Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
Add §7: what was actually built (files, functions, knobs, the keyless predicate and _precedes correction), the 11-fixture results table, the gst494 deep check (11/12, the ULUL garbage-cluster miss), and every known-broken case tagged with its fixing phase (rf-1084s column crossing = Phase 4, layout-noise filter, un-swept cap/row-band). Note the 15-vs-11 corpus-count discrepancy to reconcile. Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
§8: resume state (branch/PR target, test command, corpus-count reconcile), the headless judge recipe (unzip .dclx, parse document.xml locations, Read the page png -- replacing the viewer-only instruction the agent cannot use), the two debugging patterns that found the bugs, and the guardrails not to re-introduce (no left tiebreak, no premature DP/RO fence). Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
Sweep _LABEL_GAP_CAP_LINES / _LABEL_ROW_BAND_LINES against the 11-form corpus by replaying the pure _match_labels offline over a cap×band grid (capture once, no re-conversion). Cap is the correctness knob: 2.0 is the lower edge of the [2.0, 3.0] plateau that scores 12/12/0 on gst494; below it under-binds, at 2.5+ the corpus flips to wrong labels. Row-band is forgiving over [1.0, 2.0]. Keep both constants, no code change. Add §7.6 with the scored ground-truth method (frozen GST494_GT map, read off the raster since AcroForm has no widget→label oracle) so the verdict is rerunnable, and flip the §7.4 un-swept bullet. Reconcile the corpus count: 11 PDFs in the dir, not 15 — stale name, no convert failures. Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
At form_field time (which runs before the table model) a table's row/column keys are absorbed as TABLE children, never free-standing labels in the binding pool -- probed to be 0 in-table standalone labels on every widget-containing table across the 11-form corpus. So a widget inside a table reached OUTSIDE it for the nearest free-text and mis-bound a section header (italy SEZIONE/QUADRO, rf-1177s) or a wrong-column neighbour onto a body cell. Rule: partition the keyless widgets and the label pool by containing table (pure _table_of: smallest enclosing region by center-in-rect) and run the unchanged _match_labels per group, so binding never crosses a table boundary. A cell that carries its own key+value still binds (label in the same table); a body cell whose only nearby label is an out-of-table header stays keyless. This reverses the original §7.4 sketch (feed table cells INTO the pool): the cell grid does not exist yet at this stage, so the correct move is a boundary, not a merge. gst494 stays byte-identical (all 12 ground-truth binds intact); italy drops 11 header->body mis-binds; rf-1177s drops its header mis-bind; rf-1125s loses 3 option-label binds (accepted). Details in §9. Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
Preserve the reviewed local labels for all 946 retained native widgets, including explicit no-label and ambiguous dispositions and 503 table exclusions. Include 19 frozen page images/layout snapshots, native identities, visual review overlays, the evaluator and the design handoff. This is a reconstructed checkpoint from saved working files, not a claim of contemporaneous historical commits. The evaluator imports the optimizer introduced by the next checkpoint; the stdlib identity validator runs here. Original PDFs remain external and hash-verified; replay needs no conversion. Exclude only frozen generated snapshot JSON from the source line-count limit so recorded byte hashes remain intact. Validation: make validate; reference validator verified 11 PDF hashes, 19 pages, 970 native identities and 128 explicit table-cell memberships. Production code is unchanged. Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
Preserve the accepted offline MILP prototype, focused tests, baseline reports, coverage ablation and failure audit. Keep the pre-coverage source and change.diff alongside the accepted implementation for direct comparison. Use intersection divided by widget area for inline-clause containment at 80 percent coverage; keep detected table/cell ownership strict and retain exact page-wide native value order. The reviewed replay improves from 256 to 276 correct local labels with no previous correct labels lost. Caption fragments and missed/truncated tables remain upstream defects. Validation: 13 focused tests; all 19 pages replay optimally with 276 correct, 85 wrong and 73 unassigned labels; make validate. Generated report/diagnostic JSON is exempted from the source line-count limit to preserve saved evidence. This is an offline checkpoint, not production integration. Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
Preserve conditional question/caption configurations, structural tests, complete-assignment uncertainty checks and the raw/filtered replay reports. Include the linear-reward control and the subsequent role investigation. Reject this version for promotion: it fixes RC7190 but displaces correct GST111/F1120 captions and replaces the 21-option language group with groups seeded by caption fragments. A linear reward restores that group but merges RC7190's two questions. The README records the tradeoffs and failure evidence. Validation: 10 structural tests; frozen comparison verifies native values, order, all 503 table exclusions and reference/input hashes across 19 pages; make validate. Keep the accepted optimizer unchanged. Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
Keep multiple compatible captions available during optimization and use geometric option-row context for question proposals. Let nested groups reference their parent option's caption without consuming its text twice. Retain competing boundaries and check caption alternatives even when the question and membership agree. Conditional rewards can retain primary associations that would otherwise be pruned as costlier than abstention. Across 19 pages, fix one wrong local association without losing any previous correct labels. Recover the full language group and keep RC7190's questions separate. Sami caption selection and the outer Yes/No boundary remain unresolved; retain this version as an experiment, not production code. Validation: 30 focused tests; all 19 raw and uncertainty replays optimal; unchanged native values/order and 503 exclusions; text/name replacement leaves all group proposals unchanged; source hashes and make validate. Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
It was never intended to be committed. Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
…o codex/acroform-keying-snapshots Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
…ield stage Replace the greedy label binding with a solver that assigns each AcroForm widget its visible caption from page geometry (MILP over candidate captions). Symmetry rules make repeated rows and columns of fields read their captions from the same side, and an inline clause no longer claims checkboxes that carry their own option caption. The form_field stage now runs after table structure, and the unused checkbox_label is replaced by context_text (the aligned header of a value grid), emitted as a hint in reading order. A widget inside a detected table goes into its table cell as a rich table cell holding the field, keyed only by text printed in that same cell. Printed rules (thin painted shapes, read through the new PdfPageBackend.get_thin_shape_boxes) bound the printed cell when the detected grid does not match it; cells TableFormer left out are inserted. Field extraction is opt-in via --extract-form-fields (off by default), so default conversions are unchanged. Also place AcroForm widgets in the crop-box frame of their text, which misplaced widgets on pages with a non-zero CropBox origin, and remove the experimental keying prototype, replay script, snapshot docs and annotation data from the tree. Signed-off-by: Daniel DeAlcala <danieldealcalavalcarcel@gmail.com>
…leftovers Pass the page's widgets, layout clusters and table cells to the keying directly instead of through a snapshot copy, and clean up what the experiments left behind: - remove Assignment.objective, the null_cost/time_limit parameters and their validation, the duplicate-index guard and the unused re-exports - share the FORM-region placement between the keyed and the keyless plan, and the value/hint materialization in the reading-order model - define the rule thickness once; the backend query takes it as argument - type the keep parameter as frozenset[int] - cover acroform_sample.pdf in the e2e conversion test with reference output (DocTags excluded: its export does not support field items) Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
Signed-off-by: Christoph Auer <cau@zurich.ibm.com> # Conflicts: # docling/backend/docling_parse_backend.py # docling/cli/main.py # pyproject.toml # tests/test_backend_docling_parse.py # tests/test_e2e_conversion.py # uv.lock
Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
A checkbox block's text is now a strongly preferred candidate in the search instead of a key fixed beforehand, so its text can still key another field of the row and the checkbox keeps its group prompt. Blocks without letters are skipped, a checkbox counts as inside a block at the host coverage, and a hosted checkbox's key is located by its text. Signed-off-by: Daniel DeAlcala <danieldealcalavalcarcel@gmail.com>
This was referenced Oct 15, 2026
A checkbox in its checkbox block is again keyed by the block alone: it joins no other candidate, group prompts included, and has no blank option, so its block text is its only key and needs no prior. The text itself stays in the search under the sharing rules, so it can still key a field of the same row. This stops a group of boxed checkboxes from taking a nearby field's label as its prompt, and a boxed checkbox from being keyed by a group while its paragraph prints it a second time. A group of one boxed and one plain option now has no prompt, so the question keys the plain option; the test records this limit. Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
A boxed checkbox takes part in no association but its block's, so its other candidates are left out of the search; they also fed add_hints, and a boxed checkbox in a grid lost its column header as hint. Hints are chosen after the search and never move a key, so they now come from all candidates. Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
Signed-off-by: Christoph Auer <cau@zurich.ibm.com> # Conflicts: # docling/models/stages/reading_order/readingorder_model.py
Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
Replace the page-wide field_regions and table_fields lists with one FormFieldPrediction on PagePredictions, with items by placement route: inline host paragraph, FORM cluster, and page-wide. Write table cell items once, on the Table in table_map. Store the checkbox state as its DocItemLabel, and use an isinstance check in place of a getattr probe. Conversion output does not change. Tests read the new structures. The FORM region test runs assembly on the layout the form stage leaves, and two tests no longer use monkeypatch. Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
DanieldeAlcala
approved these changes
Oct 19, 2026
The --extract-form-fields and --to doctags check reads only options, so run it with the other option checks, before the inputs are resolved. Move its test into test_cli.py; it uses a missing source, so it never reads a file or starts a conversion. Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
Move the checkbox-block tests that check only keys into test_acroform_keying.py, where they call assign() directly, and the widget-contract test into test_backend_docling_parse.py. Merge two pairs of tests that check one rule with different inputs. Remove two tests that only locked in known wrong output. Covered form code is unchanged. Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
DanieldeAlcala
approved these changes
Oct 19, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds opt-in extraction of native PDF AcroForm widgets as format-neutral, fillable fields in
DoclingDocument. Enable it withPdfPipelineOptions(extract_form_fields=True)or--extract-form-fields. It is supported by the standard PDF pipeline and its deprecated legacy alias with the docling-parse backend. The option is off by default.A
FieldItemholds an optional field key and one or more fillableFieldValueItemchildren. A field key names values using printed text beside a widget, in its cell, or in a paragraph that contains it. Checkbox and radio option text is an option key; text that introduces several options is a shared prompt. Additional context is a field hint, stored ashint_textin predictions and emitted after the values. Here, layout label means an element class (DocItemLabel), and caption means a document caption associated with a picture, table, code block, or form.As set out in the scope decisions, this PR makes a best-effort match between native widgets and field keys in the printed form. The solver balances matching quality with computational cost. AI inference and language analysis are outside this PR's scope. Inferring widget locations or order in PDFs without AcroForm metadata is follow-up work.
The geometric solver scores candidate key-value associations using alignment, distance, intervening content, and repeated field structures. It favors consistent key positions without making them mandatory. It supports individual options, composite fields, and paragraphs that contain several widgets. Widget geometry, type, and order come from native AcroForm metadata. Native widget order determines value order within field items. Native text supplies field values. Checkbox and radio selection uses native appearance state when available; selected or unselected checkbox items are nested inside the fillable values. Push buttons and zero-size widgets are excluded. PDF tooltips and field names do not supply key text: they can be empty, unreliable, or machine identifiers. Shared PDF field names can support option grouping, but printed text supplies the field keys.
Field assignment runs after table structure. Widgets with an established table-cell assignment are emitted inside rich table cells, including grid positions omitted by table detection. Their keys come only from text in the same cell; the table's row and column headers carry the wider association. Printed strokes and thin shape boxes can refine cell ownership within detected tables. A widget inside a detected table that cannot be assigned to a cell remains outside the table without a key. Outside detected tables, aligned headers across the other axis of a value grid can supply field hints.
Text cells used as keys are removed from body text when ownership is unambiguous; unused text in partly consumed blocks remains. Shared-prompt associations leave the prompt as ordinary text. Matching printed copies of filled values are suppressed with geometry checks. Solver limits and timeouts retain free-form values without keys while retaining established table assignments. Keying exceptions fall back to unkeyed values; failed page edits restore the original layout.
This requires PDFs with native widget metadata. Scanned or flattened forms without that metadata are unsupported. Full-page OCR skips native widget extraction. Table-cell placement requires
do_table_structure=True. Raw PDF widget metadata is available on parsed pages retained withgenerate_parsed_pages=True, rather than in the document model. The service client does not expose this option, and DocTags does not support field items; the CLI rejects--extract-form-fields --to doctags. The dependency floors include docling-core 2.99.0 for field types, SciPy 1.9.0 for the solver, and docling-parse 7.22.2 for widget coordinates aligned with cropped-page text.Validation at
4b4e1feb: all 78 tests in the command below passed, including field-hint materialization. The realacroform_sample.pdfalso matched committed JSON, Markdown, and page metadata.make validatepassed in an isolated checkout. The sample is included in the PDF end-to-end test; only its unsupported DocTags comparison is skipped.