fix(pdf): preserve a line-final hyphen that does not split a word (#4043)

docling/models/stages/page_assemble/page_assemble_model.py

@@ -109,7 +109,20 @@ def _match_hyperlink(
         except ValidationError:
             return Path(best_uri)
 
-    def sanitize_text(self, lines):
+    def sanitize_text(self, lines: List[str]) -> str:
+        """Join the text lines of a cluster into a single normalized string.
+
+        Lines are joined with a space, except across a hyphen that splits a
+        word over the line break ("algo-" / "rithms"), which is dropped so
+        that the two halves close up. A hyphen only splits a word when it is
+        attached to the word it splits; one that follows whitespace is a
+        literal character -- a separator dash, a bullet marker, or a wrapped
+        hyphen-prefixed token -- so it is kept and its lines are joined like
+        any other line break.
+
+        The joined text is then normalized: a few typographic characters are
+        replaced by ASCII equivalents and ligatures are expanded.
+        """
         if len(lines) == 0:
             return ""
 
@@ -120,13 +133,18 @@ def sanitize_text(self, lines):
                 prev_words = re.findall(r"\b[\w]+\b", prev_line)
                 line_words = re.findall(r"\b[\w]+\b", line)
 
+                hyphen_attached_to_word = len(prev_line) > 1 and prev_line[-2].isalnum()
+
                 if (
-                    len(prev_words)
+                    hyphen_attached_to_word
+                    and len(prev_words)
                     and len(line_words)
                     and prev_words[-1].isalnum()
                     and line_words[0].isalnum()
                 ):
                     lines[ix] = prev_line[:-1]
+                elif not hyphen_attached_to_word:
+                    lines[ix] += " "
             else:
                 lines[ix] += " "
 

tests/data/pdf/groundtruth/2203.01017v2.doctags.txt

@@ -10,7 +10,7 @@
 <otsl><loc_258><loc_146><loc_439><loc_191><ched>1<nl></otsl>
 <text><loc_437><loc_224><loc_441><loc_228>7</text>
 <otsl><loc_258><loc_274><loc_439><loc_313><fcel>0<fcel>1 2 1<lcel><lcel><nl><fcel>3 4<fcel>5<fcel>6<fcel>7<nl><fcel>9 13<fcel>10<fcel>11<fcel>12<nl><fcel>8 2<fcel>14<fcel>15<fcel>16<nl><fcel>17<fcel>18<fcel>19<fcel>20<nl></otsl>
-<picture><loc_257><loc_143><loc_439><loc_313><caption><loc_252><loc_325><loc_445><loc_353>Figure 1: Picture of a table with subtle, complex features such as (1) multi-column headers, (2) cell with multi-row text and (3) cells with no content. Image from PubTabNet evaluation set, filename: 'PMC2944238 004 02'.</caption></picture>
+<picture><loc_257><loc_142><loc_440><loc_313><caption><loc_252><loc_325><loc_445><loc_353>Figure 1: Picture of a table with subtle, complex features such as (1) multi-column headers, (2) cell with multi-row text and (3) cells with no content. Image from PubTabNet evaluation set, filename: 'PMC2944238 004 02'.</caption></picture>
 <text><loc_252><loc_369><loc_445><loc_420>Recently, significant progress has been made with vision based approaches to extract tables in documents. For the sake of completeness, the issue of table extraction from documents is typically decomposed into two separate challenges, i.e. (1) finding the location of the table(s) on a document-page and (2) finding the structure of a given table in the document.</text>
 <text><loc_252><loc_422><loc_445><loc_450><loc_41><loc_48><loc_234><loc_61>The first problem is called table-location and has been previously addressed [30, 38, 19, 21, 23, 26, 8] with stateof-the-art object-detection networks (e.g. YOLO and later on Mask-RCNN [9]). For all practical purposes, it can be considered as a solved problem, given enough ground-truth data to train on.</text>
 <page_footer><loc_241><loc_464><loc_245><loc_469>1</page_footer>
@@ -151,7 +151,7 @@
 <list_item><loc_41><loc_376><loc_234><loc_401>Quanzeng You, Hailin Jin, Zhaowen Wang, Chen Fang, and Jiebo Luo. Image captioning with semantic attention. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 4651-4659, 2016. 4</list_item>
 <list_item><loc_41><loc_403><loc_234><loc_436>Xinyi Zheng, Doug Burdick, Lucian Popa, Peter Zhong, and Nancy Xin Ru Wang. Global table extractor (gte): A framework for joint table identification and cell structure recognition using visual context. Winter Conference for Applications in Computer Vision (WACV) , 2021. 2, 3</list_item>
 <list_item><loc_41><loc_438><loc_234><loc_450>Xu Zhong, Elaheh ShafieiBavani, and Antonio Jimeno Yepes. Image-based table recognition: Data, model,</list_item>
-<list_item><loc_269><loc_48><loc_445><loc_74>and evaluation. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm, editors, Computer Vision ECCV 2020 , pages 564-580, Cham, 2020. Springer International Publishing. 2, 3, 7</list_item>
+<list_item><loc_269><loc_48><loc_445><loc_74>and evaluation. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm, editors, Computer Vision - ECCV 2020 , pages 564-580, Cham, 2020. Springer International Publishing. 2, 3, 7</list_item>
 <list_item><loc_252><loc_76><loc_445><loc_102>Xu Zhong, Jianbin Tang, and Antonio Jimeno Yepes. Publaynet: Largest dataset ever for document layout analysis. In 2019 International Conference on Document Analysis and Recognition (ICDAR) , pages 1015-1022, 2019. 1</list_item>
 </unordered_list>
 <page_footer><loc_239><loc_464><loc_247><loc_469>10</page_footer>

tests/data/pdf/groundtruth/2203.01017v2.json

@@ -7221,11 +7221,11 @@
           },
           "charspan": [
             0,
-            55
+            56
           ]
         }
       ],
-      "orig": "l box = \u03bb iou l iou + \u03bb l 1 l = \u03bbl s +(1 -\u03bb ) l box (1)",
+      "orig": "l box = \u03bb iou l iou + \u03bb l 1 l = \u03bbl s +(1 - \u03bb ) l box (1)",
       "text": ""
     },
     {
@@ -7671,11 +7671,11 @@
           },
           "charspan": [
             0,
-            74
+            75
           ]
         }
       ],
-      "orig": "TEDS( T a , T b ) = 1 -EditDist ( T a , T b ) max( | T a | , | T b | ) (3)",
+      "orig": "TEDS( T a , T b ) = 1 - EditDist ( T a , T b ) max( | T a | , | T b | ) (3)",
       "text": ""
     },
     {
@@ -13763,12 +13763,12 @@
           },
           "charspan": [
             0,
-            192
+            194
           ]
         }
       ],
-      "orig": "and evaluation. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm, editors, Computer Vision ECCV 2020 , pages 564-580, Cham, 2020. Springer International Publishing. 2, 3, 7",
-      "text": "and evaluation. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm, editors, Computer Vision ECCV 2020 , pages 564-580, Cham, 2020. Springer International Publishing. 2, 3, 7",
+      "orig": "and evaluation. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm, editors, Computer Vision - ECCV 2020 , pages 564-580, Cham, 2020. Springer International Publishing. 2, 3, 7",
+      "text": "and evaluation. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm, editors, Computer Vision - ECCV 2020 , pages 564-580, Cham, 2020. Springer International Publishing. 2, 3, 7",
       "enumerated": false,
       "marker": ""
     },
@@ -16295,11 +16295,11 @@
           },
           "charspan": [
             0,
-            64
+            65
           ]
         }
       ],
-      "orig": "alignment = arg min c { D c } D c = max { x c } -min { x c } (4)",
+      "orig": "alignment = arg min c { D c } D c = max { x c } - min { x c } (4)",
       "text": ""
     },
     {
@@ -17294,9 +17294,9 @@
           "page_no": 1,
           "bbox": {
             "l": 314.77,
-            "t": 566.19,
-            "r": 537.93,
-            "b": 295.62,
+            "t": 566.39,
+            "r": 537.95,
+            "b": 295.61,
             "coord_origin": "BOTTOMLEFT"
           },
           "charspan": [
@@ -18854,7 +18854,7 @@
           "bbox": {
             "l": 315.46,
             "t": 561.46,
-            "r": 537.05,
+            "r": 537.04,
             "b": 489.47,
             "coord_origin": "BOTTOMLEFT"
           },
@@ -18932,10 +18932,10 @@
         {
           "page_no": 1,
           "bbox": {
-            "l": 315.21,
-            "t": 358.2,
+            "l": 315.2,
+            "t": 358.21,
             "r": 537.11,
-            "b": 295.44,
+            "b": 295.43,
             "coord_origin": "BOTTOMLEFT"
           },
           "charspan": [

tests/data/pdf/groundtruth/2203.01017v2.md

@@ -270,7 +270,7 @@ end object detection with transformers. In Andrea Vedaldi, Horst Bischof, Thomas
 - [35] Quanzeng You, Hailin Jin, Zhaowen Wang, Chen Fang, and Jiebo Luo. Image captioning with semantic attention. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 4651-4659, 2016. 4
 - [36] Xinyi Zheng, Doug Burdick, Lucian Popa, Peter Zhong, and Nancy Xin Ru Wang. Global table extractor (gte): A framework for joint table identification and cell structure recognition using visual context. Winter Conference for Applications in Computer Vision (WACV) , 2021. 2, 3
 - [37] Xu Zhong, Elaheh ShafieiBavani, and Antonio Jimeno Yepes. Image-based table recognition: Data, model,
-- and evaluation. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm, editors, Computer Vision ECCV 2020 , pages 564-580, Cham, 2020. Springer International Publishing. 2, 3, 7
+- and evaluation. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm, editors, Computer Vision - ECCV 2020 , pages 564-580, Cham, 2020. Springer International Publishing. 2, 3, 7
 - [38] Xu Zhong, Jianbin Tang, and Antonio Jimeno Yepes. Publaynet: Largest dataset ever for document layout analysis. In 2019 International Conference on Document Analysis and Recognition (ICDAR) , pages 1015-1022, 2019. 1
 
 ## TableFormer: Table Structure Understanding with Transformers

tests/data/pdf/groundtruth/2206.01062.doctags.txt

@@ -114,7 +114,7 @@
 <text><loc_259><loc_182><loc_456><loc_200>To date, there is still a significant gap between human and ML accuracy on the layout interpretation task, and we hope that this work will inspire the research community to close that gap.</text>
 <section_header_level_1><loc_260><loc_212><loc_316><loc_218>REFERENCES</section_header_level_1>
 <ordered_list><list_item><loc_262><loc_220><loc_456><loc_234>Max Göbel, Tamir Hassan, Ermelinda Oro, and Giorgio Orsi. Icdar 2013 table competition. In 2013 12th International Conference on Document Analysis and Recognition , pages 1449-1453, 2013.</list_item>
-<list_item><loc_262><loc_235><loc_457><loc_254>Christian Clausner, Apostolos Antonacopoulos, and Stefan Pletschacher. Icdar2017 competition on recognition of documents with complex layouts rdcl2017. In 2017 14th IAPR International Conference on Document Analysis and Recognition (ICDAR) , volume 01, pages 1404-1410, 2017.</list_item>
+<list_item><loc_262><loc_235><loc_457><loc_254>Christian Clausner, Apostolos Antonacopoulos, and Stefan Pletschacher. Icdar2017 competition on recognition of documents with complex layouts - rdcl2017. In 2017 14th IAPR International Conference on Document Analysis and Recognition (ICDAR) , volume 01, pages 1404-1410, 2017.</list_item>
 <list_item><loc_262><loc_256><loc_456><loc_269>Hervé Déjean, Jean-Luc Meunier, Liangcai Gao, Yilun Huang, Yu Fang, Florian Kleber, and Eva-Maria Lang. ICDAR 2019 Competition on Table Detection and Recognition (cTDaR), April 2019. http://sac.founderit.com/.</list_item>
 <list_item><loc_262><loc_271><loc_457><loc_290>Antonio Jimeno Yepes, Peter Zhong, and Douglas Burdick. Competition on scientific literature parsing. In Proceedings of the International Conference on Document Analysis and Recognition , ICDAR, pages 605-617. LNCS 12824, SpringerVerlag, sep 2021.</list_item>
 <list_item><loc_262><loc_291><loc_457><loc_310>Logan Markewich, Hao Zhang, Yubin Xing, Navid Lambert-Shirzad, Jiang Zhexin, Roy Lee, Zhi Li, and Seok-Bum Ko. Segmentation for document layout analysis: not dead yet. International Journal on Document Analysis and Recognition (IJDAR) , pages 1-11, 01 2022.</list_item>

tests/data/pdf/groundtruth/2206.01062.json

@@ -12874,12 +12874,12 @@
           },
           "charspan": [
             0,
-            279
+            281
           ]
         }
       ],
-      "orig": "[2] Christian Clausner, Apostolos Antonacopoulos, and Stefan Pletschacher. Icdar2017 competition on recognition of documents with complex layouts rdcl2017. In 2017 14th IAPR International Conference on Document Analysis and Recognition (ICDAR) , volume 01, pages 1404-1410, 2017.",
-      "text": "Christian Clausner, Apostolos Antonacopoulos, and Stefan Pletschacher. Icdar2017 competition on recognition of documents with complex layouts rdcl2017. In 2017 14th IAPR International Conference on Document Analysis and Recognition (ICDAR) , volume 01, pages 1404-1410, 2017.",
+      "orig": "[2] Christian Clausner, Apostolos Antonacopoulos, and Stefan Pletschacher. Icdar2017 competition on recognition of documents with complex layouts - rdcl2017. In 2017 14th IAPR International Conference on Document Analysis and Recognition (ICDAR) , volume 01, pages 1404-1410, 2017.",
+      "text": "Christian Clausner, Apostolos Antonacopoulos, and Stefan Pletschacher. Icdar2017 competition on recognition of documents with complex layouts - rdcl2017. In 2017 14th IAPR International Conference on Document Analysis and Recognition (ICDAR) , volume 01, pages 1404-1410, 2017.",
       "enumerated": true,
       "marker": "[2]"
     },
@@ -15121,10 +15121,10 @@
         {
           "page_no": 1,
           "bbox": {
-            "l": 322.79,
-            "t": 543.26,
-            "r": 554.34,
-            "b": 266.09,
+            "l": 322.78,
+            "t": 543.27,
+            "r": 554.33,
+            "b": 266.1,
             "coord_origin": "BOTTOMLEFT"
           },
           "charspan": [

tests/data/pdf/groundtruth/2206.01062.md

@@ -269,7 +269,7 @@ To date, there is still a significant gap between human and ML accuracy on the l
 ## REFERENCES
 
 - [1] Max Göbel, Tamir Hassan, Ermelinda Oro, and Giorgio Orsi. Icdar 2013 table competition. In 2013 12th International Conference on Document Analysis and Recognition , pages 1449-1453, 2013.
-- [2] Christian Clausner, Apostolos Antonacopoulos, and Stefan Pletschacher. Icdar2017 competition on recognition of documents with complex layouts rdcl2017. In 2017 14th IAPR International Conference on Document Analysis and Recognition (ICDAR) , volume 01, pages 1404-1410, 2017.
+- [2] Christian Clausner, Apostolos Antonacopoulos, and Stefan Pletschacher. Icdar2017 competition on recognition of documents with complex layouts - rdcl2017. In 2017 14th IAPR International Conference on Document Analysis and Recognition (ICDAR) , volume 01, pages 1404-1410, 2017.
 - [3] Hervé Déjean, Jean-Luc Meunier, Liangcai Gao, Yilun Huang, Yu Fang, Florian Kleber, and Eva-Maria Lang. ICDAR 2019 Competition on Table Detection and Recognition (cTDaR), April 2019. http://sac.founderit.com/.
 - [4] Antonio Jimeno Yepes, Peter Zhong, and Douglas Burdick. Competition on scientific literature parsing. In Proceedings of the International Conference on Document Analysis and Recognition , ICDAR, pages 605-617. LNCS 12824, SpringerVerlag, sep 2021.
 - [5] Logan Markewich, Hao Zhang, Yubin Xing, Navid Lambert-Shirzad, Jiang Zhexin, Roy Lee, Zhi Li, and Seok-Bum Ko. Segmentation for document layout analysis: not dead yet. International Journal on Document Analysis and Recognition (IJDAR) , pages 1-11, 01 2022.

tests/data/pdf/groundtruth/2305.03393v1.doctags.txt

@@ -1,6 +1,6 @@
 <doctag><page_header><loc_15><loc_132><loc_30><loc_350>arXiv:2305.03393v1  [cs.CV]  5 May 2023</page_header>
 <section_header_level_1><loc_110><loc_73><loc_393><loc_92>Optimized Table Tokenization for Table Structure Recognition</section_header_level_1>
-<text><loc_114><loc_107><loc_389><loc_126>Maksym Lysak [0000 -0002 -3723 -6960] , Ahmed Nassar [0000 -0002 -9468 -0822] , Nikolaos Livathinos [0000 -0001 -8513 -3491] , Christoph Auer [0000 -0001 -5761 -0422] , [0000 -0002 -8088 -0823]</text>
+<text><loc_114><loc_107><loc_389><loc_126>Maksym Lysak [0000 - 0002 - 3723 - 6960] , Ahmed Nassar [0000 - 0002 - 9468 - 0822] , Nikolaos Livathinos [0000 - 0001 - 8513 - 3491] , Christoph Auer [0000 - 0001 - 5761 - 0422] , [0000 - 0002 - 8088 - 0823]</text>
 <text><loc_188><loc_123><loc_244><loc_129>and Peter Staar</text>
 <text><loc_228><loc_136><loc_275><loc_143>IBM Research</text>
 <text><loc_182><loc_144><loc_321><loc_149>{mly,ahn,nli,cau,taa}@zurich.ibm.com</text>
@@ -46,11 +46,11 @@
 <section_header_level_1><loc_110><loc_294><loc_214><loc_300>4.1 Language Definition</section_header_level_1>
 <text><loc_110><loc_309><loc_393><loc_329>In Figure 3, we illustrate how the OTSL is defined. In essence, the OTSL defines only 5 tokens that directly describe a tabular structure based on an atomic 2D grid.</text>
 <text><loc_122><loc_332><loc_334><loc_337>The OTSL vocabulary is comprised of the following tokens:</text>
-<unordered_list><list_item><loc_115><loc_346><loc_376><loc_352>-"C" cell a new table cell that either has or does not have cell content</list_item>
-<list_item><loc_115><loc_354><loc_393><loc_367>-"L" cell left-looking cell , merging with the left neighbor cell to create a span</list_item>
-<list_item><loc_115><loc_369><loc_393><loc_382>-"U" cell up-looking cell , merging with the upper neighbor cell to create a span</list_item>
-<list_item><loc_115><loc_385><loc_371><loc_390>-"X" cell cross cell , to merge with both left and upper neighbor cells</list_item>
-<list_item><loc_115><loc_393><loc_268><loc_398>-"NL" new-line , switch to the next row.</list_item>
+<unordered_list><list_item><loc_115><loc_346><loc_376><loc_352>"C" cell - a new table cell that either has or does not have cell content</list_item>
+<list_item><loc_115><loc_354><loc_393><loc_367>"L" cell - left-looking cell , merging with the left neighbor cell to create a span</list_item>
+<list_item><loc_115><loc_369><loc_393><loc_382>"U" cell - up-looking cell , merging with the upper neighbor cell to create a span</list_item>
+<list_item><loc_115><loc_385><loc_371><loc_390>"X" cell - cross cell , to merge with both left and upper neighbor cells</list_item>
+<list_item><loc_115><loc_393><loc_268><loc_398>"NL" - new-line , switch to the next row.</list_item>
 </unordered_list>
 <text><loc_110><loc_407><loc_393><loc_420>A notable attribute of OTSL is that it has the capability of achieving lossless conversion to HTML.</text>
 <page_break>