Skip to content

feat: avoid padded-grid allocation in table exports - #1

Merged
npkanaka merged 2 commits into
mainfrom
fix/record-table-serve
Oct 9, 2026
Merged

npkanaka merged 2 commits into
mainfrom
fix/record-table-serve

Conversation

@npkanaka

@npkanaka npkanaka commented Oct 9, 2026

Copy link
Copy Markdown
Collaborator

What

Add an opt-in, record-oriented Markdown table exporter and a CPU-only serving image built from this repository. Table exports retain actual values without allocating a padded rectangular grid.

Why

Wide, sparse tables can use excessive memory during export. Iterating existing cells avoids padding missing positions while retaining explicit empty fields, coordinates, header flags and merged ranges.

Thought

Keep the structured document model and other export formats unchanged. Enable the exporter through a process-local startup factory so normal, worker and reload processes use the same behavior. Build the converter and adapter from the checkout, with a pinned runtime and installed-source verification.

Alternatives considered

  • Converting CSV tables into plain text would discard the structured table model.
  • Changing the SDK's default exporter globally would affect callers that did not opt in. A dedicated serving integration keeps that boundary explicit.

Blindspots

  • Local execution covers native ARM64 and emulated AMD64. Native execution on both architectures is configured in CI but has not yet been verified remotely.
  • End-to-end conversion checks cover CSV and HTML, not PDF model behavior or concurrent large-upload capacity.
  • Exported Markdown can be larger than a compact table. Downstream consumers still need appropriate storage and request-size limits.
  • Functional tests do not establish vulnerability remediation or production readiness.
  • GHCR publication and package permissions have not been exercised. Publication rebuilds the tested source revision rather than promoting identical local test-image binaries.

ELI5

Read the filled squares in a spreadsheet instead of creating every empty square in a huge rectangle.

Test plan

  • 19 unit tests, including no-grid access, caption placement, rich-cell heading behavior, merged cells and unchanged document JSON.
  • CSV and HTML HTTP conversion in default, worker and reload modes on native ARM64 and emulated AMD64.
  • Large synthetic CSV conversion in all three modes on ARM64, comparing all returned values and record labels with the source.
  • Worker identity, unexpected restart and OOM-counter checks.
  • Installed converter version/source verification and unfiltered dependency checks.
  • Ruff lint/format, Actionlint and staged whitespace checks.
  • Confirm native AMD64 and ARM64 CI checks.
  • Verify GHCR publication and the resulting multi-platform manifest after review and merge.

See integrations/stackgen_serve/README.md for build commands and limits.

@npkanaka
npkanaka marked this pull request as ready for review October 9, 2026 19:48
@github-actions

github-actions Bot commented Oct 9, 2026 •

Copy link
Copy Markdown

AI triage (advisory)

Checked commit c1f24ea890. The analysis read the diff only. It did not run code from this pull request.

Warning

The AI analysis did not finish. Only the deterministic checks are shown.

CI safety: needs maintainer care before you approve CI.

  • .github/workflows/stackgen-serve.yml: changes CI workflows, actions, or CI scripts
  • integrations/stackgen_serve/build.sh: adds or changes a shell script
  • integrations/stackgen_serve/test-image.sh: adds or changes a shell script

The ai:* labels are advisory. They do not approve CI and they do not add tests:full. A new push runs this check again.

@npkanaka npkanaka changed the title Avoid padded-grid allocation in table exports feat: avoid padded-grid allocation in table exports Oct 9, 2026
@npkanaka
npkanaka merged commit ead4de6 into main Oct 9, 2026
53 checks passed
@sks sks mentioned this pull request Oct 12, 2026
2 tasks done
sks added a commit that referenced this pull request Oct 12, 2026
## Summary

- Add opt-in record-oriented table output for Markdown exports. The
default Markdown table format does not change.
- Support `--markdown-table-format records` in `convert` and
`convert-remote`. Remote conversion still applies the option only when
writing Markdown locally.
- Provide `export_to_record_markdown()` and `save_as_record_markdown()`
for packaged SDK users. Keep image modes and other output formats
unchanged.
- Remove the fork-specific Serve integration from PR #1. The serializer
does not replace a process-wide Core serializer.

This branch is based on the current `stackgenhq/main`. [Draft PR
#2](#2) contains the initial
serializer. The [branch
comparison](feat/record-table-serializer...feat/record-table-export-dmain)
now shows only the 11-file opt-in addition, without unrelated upstream
commits.

## Checks

- Selected tests: 164 passed, 2 deselected (`test_cli_convert`), 91
warnings.
- `make validate` passed on the 11-file opt-in changeset.
- `git diff --check` passed.
- Full repository test suite and `make check` were not run for this PR.

This export option does not address conversion memory use or add a
Docling Serve-wide option. A general Markdown serializer API remains a
`docling-core` design decision.

- [x] Documentation updated
- [x] Tests added
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant