HybridChunker._split_by_doc_items re-serializes an InlineGroup's runs independently, breaking inline formatting #770

HierarchicalChunker serializes an InlineGroup as one unit. A paragraph with inline markup chunks as Install with `pip install x` first.. HybridChunker.chunk then passes every chunk through _split_by_doc_items, which rebuilds each window with _make_chunk_from_doc_items. That method keeps the original text only when the chunk has a single doc_item. Otherwise it builds the text as

self.delim.join(
    res_text
    for doc_item in doc_items
    if (res_text := doc_serializer.serialize(item=doc_item).text)
    and not isinstance(doc_item, TitleItem | SectionHeaderItem)
)

The doc_items of an inline-group chunk are its runs. Each run is serialized as a standalone item. A TextItem becomes its own line and a CodeItem becomes a fenced code block. The rebuild happens whether or not the chunk needs splitting, because _split_by_doc_items always reconstructs the window. It happens before peer merging, and the corruption occurs with merge_peers=False. The TODO: merging should ideally be done by the serializer above the join marks the spot.

Reproduced on docling-core 2.96.0 with the default tokenizer. The code is unchanged on main. This is distinct from #371 and PR #693, which concern the whitespace between runs when a group is serialized as one unit. Here the runs are not serialized as one unit at all.

Reproduction

from docling_core.types.doc.document import DoclingDocument
from docling_core.types.doc.labels import DocItemLabel
from docling_core.transforms.chunker.hierarchical_chunker import HierarchicalChunker
from docling_core.transforms.chunker.hybrid_chunker import HybridChunker

doc = DoclingDocument(name="t")
doc.add_title(text="Sample")
heading = doc.add_heading(text="", level=1)
group = doc.add_inline_group(parent=heading)
doc.add_text(label=DocItemLabel.TEXT, text="Release", parent=group, hyperlink="https://example.com/releases")
doc.add_text(label=DocItemLabel.TEXT, text="notes", parent=group)
para = doc.add_inline_group()
doc.add_text(label=DocItemLabel.TEXT, text="Install with", parent=para)
doc.add_code(text="pip install x", parent=para)
doc.add_text(label=DocItemLabel.TEXT, text="first.", parent=para)

for c in HierarchicalChunker().chunk(doc):
    print("hierarchical:      ", repr(c.text))
for c in HybridChunker(merge_peers=False).chunk(doc):
    print("hybrid, no merging:", repr(c.text))
for c in HybridChunker().chunk(doc):
    print("hybrid, default:   ", repr(c.text))

Output on 2.96.0:

hierarchical:       '[Release](https://example.com/releases) notes'
hierarchical:       'Install with `pip install x` first.'
hybrid, no merging: '[Release](https://example.com/releases)\nnotes'
hybrid, no merging: 'Install with\n```\npip install x\n```\nfirst.'
hybrid, default:    '[Release](https://example.com/releases)\nnotes\nInstall with\n```\npip install x\n```\nfirst.'

A real document shows the same. docling 2.124.0's Markdown backend produces this shape for a paragraph mixing ordinary text with a code span. A paragraph that is only a code span is a bare CodeItem. A changelog converted and chunked with HybridChunker carries By default,\n```\nhaiku.rag\n```\nuses the configured embedder. where the source has By default, `haiku.rag` uses the configured embedder.

Expected

A rebuilt window should render its items the way the serializer renders them together. The hierarchical chunks already contain that text. With this fixed alone:

hybrid, no merging: '[Release](https://example.com/releases) notes'
hybrid, no merging: 'Install with `pip install x` first.'
hybrid, default:    '[Release](https://example.com/releases) notes\nInstall with `pip install x` first.'

The heading's runs appearing in the body at all is the separate meta.headings issue, #769. With both fixed, the first line disappears from the body and Release notes appears in the metadata.