Skip to content

fix(examples): record the document-to-markdown run against SIE Cloud #280

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
svonava merged 1 commit into from
Sep 17, 2026
Merged

fix(examples): record the document-to-markdown run against SIE Cloud #280

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 15 additions & 3 deletions examples/document-to-markdown/.gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,18 @@
.env
.venv/

# The source PDFs stay fetch-only. Two of the four are investor documents whose
# rights notes in config.yaml say not to redistribute the complete PDF, and the
# Docling paper is left fetch-only in deference to the same note even though it
# is CC BY 4.0. data/manifest.json below pins every URL, digest and byte length,
# which is what a reader needs to confirm they fetched the same bytes.
data/pdfs/
data/manifest.json
runs/*
!runs/.gitkeep

# data/manifest.json and runs/ are deliberately NOT ignored.
#
# This example was built to commit nothing: it fetched at run time and excluded
# every artifact it produced, so its published result -- "all 25 checks passed"
# -- had no record behind it that anyone could check. That was the defect, not
# an oversight, and un-ignoring these is the fix. Committing artifacts is
# already house style here; examples/threat-report-attck-mapper commits its
# dataset under data/cache/.
181 changes: 170 additions & 11 deletions examples/document-to-markdown/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,15 +29,85 @@ not pretend a form has one useful reading order.

## Verified result

We ran the example on an NVIDIA L4 through a local SIE endpoint on July 24,
2026. All 25 checks passed:
Recorded against SIE Cloud, `https://api.superlinked.com`, on September 16,
2026. All 25 checks passed. The run is committed under
[`runs/cloud-20260916/`](runs/cloud-20260916) — every request, response, model
revision and timing — so these numbers can be checked rather than taken on
trust.

| Document | Checks | SIE latency |
| Document | Checks | Latency, recorded run |
|---|---:|---:|
| NVIDIA Q4 FY2025 CFO commentary | 6/6 | 50.3 s |
| SiriusPoint Q1 2025 investor presentation | 6/6 | 16.3 s |
| Docling technical report | 6/6 | 11.7 s |
| FEMA proof-of-loss form | 7/7 | 10.0 s |
| NVIDIA Q4 FY2025 CFO commentary | 6/6 | 9.2 s |
| SiriusPoint Q1 2025 investor presentation | 6/6 | 12.2 s |
| Docling technical report | 6/6 | 6.7 s |
| FEMA proof-of-loss form | 7/7 | 1.3 s |

### All three runs

The session made three complete runs of the same four documents against the
same endpoint. All three passed, and all three produced **byte-identical
Markdown** — the conversion is deterministic here, so this is reproducible
rather than merely repeated. Every timing is below, because publishing one
run's numbers without the others invites you to read a single measurement as a
typical one.

| Document | Run 1 | Run 2 | Run 3 (recorded) |
|---|---:|---:|---:|
| NVIDIA Q4 FY2025 CFO commentary | 24.7 s | 9.8 s | 9.2 s |
| SiriusPoint Q1 2025 investor presentation | 12.9 s | 11.7 s | 12.2 s |
| Docling technical report | 7.3 s | 7.2 s | 6.7 s |
| FEMA proof-of-loss form | 1.4 s | 1.3 s | 1.3 s |
| **Total** | **46.3 s** | **30.1 s** | **29.3 s** |

Totals are summed from the unrounded measurements, so adding the rounded rows
above can land a tenth of a second away — run 3's rows come to 29.4 s against a
true total of 29.3 s. The exact milliseconds are in each run's manifest.

Run 3 is the committed one because it is the only run produced by the code in
this commit: runs 1 and 2 predate `convert.py` recording a per-call model
revision, so they carry no `calls.json` and `verify-run` cannot check them.
That is the reason, and it is worth being blunt about what it does not explain.
**Run 3 is also the fastest of the three overall, and fastest on three of the
four documents.** It was not chosen for that.

[`runs/repeat-runs/repeat-runs.json`](runs/repeat-runs/repeat-runs.json) keeps
runs 1 and 2 — their timings and the SHA-256 of every file they produced — but
not their Markdown, so the corpus is redistributed once rather than three
times. To check the byte-identical claim yourself, hash
`runs/cloud-20260916/markdown/<slug>.md` and compare.

Read any of these as single measurements on shared Cloud hardware we do not
control, not as a benchmark. The model was not resident when the session began,
and that first sequence paid about 15 seconds of provisioning on its first
document alone — 24.7 s against 9.8 s next time — while the other three moved
by under 1.3 s across all three runs.

The Markdown, though, did not move at all. `convert.py` changed between run 2
and run 3, to record the per-call model revision, and all four digests are the
same before and after. A change that adds provenance should leave the thing it
describes untouched, and this one did.

| Document | SHA-256 of the converted Markdown, identical in all three runs |
|---|---|
| NVIDIA Q4 FY2025 CFO commentary | `2c64f61d65417edf…` |
| SiriusPoint Q1 2025 investor presentation | `153064274f4e3bfc…` |
| Docling technical report | `47b058fe6676cbea…` |
| FEMA proof-of-loss form | `75520494bf872ebb…` |

### Verify without running

The whole bundle verifies offline, with no API key and no network:

```bash
uv run verify-run runs/cloud-20260916
```

It recomputes the 25 from the recorded check arrays rather than reading a
stored total, checks every digest, and asserts that each scored Markdown file is
exactly the Markdown its recorded API response returned. That last check exists
because `eval-documents` reads `runs/<id>/markdown/<slug>.md` and never opens
the response beside it, so on its own it would be scoring a file this harness
wrote. Binding the two was added after running the example exposed the gap.

## Run it

Expand Down Expand Up @@ -73,13 +143,102 @@ SIE_API_KEY=...
config.yaml source URLs, model, and acceptance checks
fixtures/SOURCES.md rights and attribution notes
data/pdfs/ fetched PDFs, ignored by git
data/manifest.json acquired checksums, ignored by git
runs/<run-id>/manifest.json endpoint, model, latency, and output paths
runs/<run-id>/raw/*.json complete SIE responses
runs/<run-id>/markdown/*.md exported Markdown
data/manifest.json source URL, rights, byte length and SHA-256 per PDF
runs/<run-id>/manifest.json endpoint, model revisions, timings, and digests
runs/<run-id>/calls.json every call in one file: request, response,
status, headers, timing, model revision
runs/<run-id>/payloads/*.json large unscored response members, by digest
runs/<run-id>/markdown/*.md exported Markdown, the text the checks score
runs/<run-id>/evaluation.json exact pass and failure details
runs/repeat-runs/ earlier complete runs, kept for the
byte-identical claim above
```

The source PDFs stay fetch-only: two of the four are investor documents whose
rights notes say not to redistribute the complete file. `data/manifest.json` is
committed instead, pinning each URL, byte length and SHA-256, which is what a
reader needs to confirm they fetched the same bytes this run scored.

One call is one entry in `calls.json` rather than a file of its own. Members
that nothing scores move to `payloads/` and are referenced by digest — for
Docling that is `data.document`, between 85 and 98 percent of every response.
Those payloads are digested over their stored bytes rather than canonically,
because they carry `origin.binary_hash`, a 64-bit integer that JavaScript
cannot parse without silently changing it.

Two kinds of digest appear in a run, and which is which matters if you go to
check one:

| Digest | Taken over |
| --- | --- |
| `manifest_sha256`, `calls.sha256`, `entry_sha256`, `scored_markdown.response_markdown_sha256` | RFC 8785 canonical encoding of the parsed value, so reformatting a copy does not change it |
| `$payload.sha256`, `scored_markdown.sha256`, `document_sha256`, `source_sha256`, and `sha256` in `data/manifest.json` | the recorded file bytes, reproducible with `shasum -a 256` |

The split is not arbitrary. Content that is compared against a reformatted copy
elsewhere is hashed canonically; content that is a file a reader will hash
themselves is hashed as bytes.

One consequence worth knowing before you edit anything. The note describing this
scheme lives *inside* `calls.json`, so correcting that note changes
`calls.sha256` and `manifest_sha256` — while every `entry_sha256` stays fixed,
because an entry holds only its own request, response, timing and model
revision. That is the intended shape: the entry digests are what pin the
evidence, and they cannot move without a request or a response moving. If you
change the prose and the two document-level digests move while the four entry
digests do not, nothing about the run has changed. If an entry digest moves, the
run has.

## What the conversion gets wrong

All 25 checks pass. The tables are still not structurally reliable, and those
two facts are both true because the checks test different things than you might
assume. Everything below is in `runs/cloud-20260916/markdown/` to read.

**The tables do not survive as tables.** There is not one structurally clean
table in this corpus — I checked all fifteen. The behaviour is consistent:
Docling expands a spanning header across every cell it covers instead of
merging it, and it does not separate tables that are stacked on a page.

The NVIDIA output shows both at once. Lines 42 to 72 look like one 31-row
Markdown table. They are four separate PDF tables welded together, with each
section label expanded across all four cells acting as the seam:

Lines 42, 45, 50, 51 and 53 of that file, verbatim:

```text
| ($ in millions, except earnings per share) | FY25 | FY24 | Y/Y |
| Gross margin | 75.0 % | 72.7 % | Up 2.3 pts |
| Non-GAAP | Non-GAAP | Non-GAAP | Non-GAAP |
| ($ in millions, except earnings per share) | FY25 | FY24 | Y/Y |
| Gross margin | 75.5 % | 73.8 % | Up 1.7 pts |
```

GAAP and non-GAAP figures for the same line item end up in one table under one
header, with nothing but a repeated word between them. The Q4 table above it
welds four PDF tables the same way. SiriusPoint's header repeats
`Financial Highlights` across all four columns and its `Q1'24` column comes
back twice. The Docling paper's own benchmark table goes further and collapses
multi-row cells into space-joined values — `4 16`, `177 s 167 s` — while a
second table flattens an entire per-class accuracy table into two cells, one
holding every label and one holding every value.

**Three smaller behaviours.** Fenced code blocks are flattened onto one line,
so the four-statement Python example in the converted Docling paper would not
run as printed. Line-break hyphens are closed up, so `MIT-licensed` returns as
`MITlicensed`. And on the FEMA form every section heading is emitted together
near the top while its fields appear much later — `TYPE OF PROOF OF LOSS` at
line 7, its six choices from line 57. The content survives; the grouping does
not.

**Why the checks pass anyway, and what that says about the checks.** They test
exact facts, section order, and that tables are present — not that a table is
the right shape. `_table_count` counts contiguous pipe blocks, so the four
welded NVIDIA tables count as one, and `7 found, 4 required` passes honestly
while saying nothing about structure. That is a limitation of this evaluation,
not a detail about the model, and it is worth knowing before you copy the
approach: a conversion harness that checks facts and order will not tell you
your tables are wrong.

## Honest scope

This example measures whether the converted structure keeps the facts and order
Expand Down
53 changes: 53 additions & 0 deletions examples/document-to-markdown/data/manifest.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,53 @@
{
"fetched_at": "2026-09-16T06:31:20.832685+00:00",
"documents": [
{
"slug": "nvidia-cfo-commentary",
"title": "NVIDIA Q4 FY2025 CFO Commentary",
"publisher": "NVIDIA",
"file_name": "nvidia-q4-fy2025-cfo-commentary.pdf",
"url": "https://investor.nvidia.com/files/doc_financials/2025/Q425/Q4FY25-CFO-Commentary.pdf",
"source_page": "https://investor.nvidia.com/financial-info/quarterly-results/default.aspx",
"rights": "Public investor document. Fetch at run time; do not redistribute the complete PDF.",
"retrieval": "publisher",
"bytes": 82336,
"sha256": "842a50961f0466d50584f8017200ab72c5a86c7438f0560cefb0506442149ce7"
},
{
"slug": "siriuspoint-investor-deck",
"title": "SiriusPoint Q1 2025 Investor Presentation",
"publisher": "SiriusPoint",
"file_name": "siriuspoint-q1-2025-investor-presentation.pdf",
"url": "https://s27.q4cdn.com/660241321/files/doc_financials/2025/q1/Q1-2025-SPNT-Investor-Presentation_vFinal.pdf",
"source_page": "https://investors.siriuspt.com/financials/quarterly-results/default.aspx",
"rights": "Public investor presentation. Fetch at run time; do not redistribute the complete PDF.",
"retrieval": "publisher",
"bytes": 2029292,
"sha256": "aab5e5c3cd2de163ca39dba485d0d2d753254854ae543d9c265a91babc32b34f"
},
{
"slug": "docling-paper",
"title": "Docling Technical Report",
"publisher": "Docling project authors",
"file_name": "docling-technical-report.pdf",
"url": "https://arxiv.org/pdf/2408.09869",
"source_page": "https://arxiv.org/abs/2408.09869",
"rights": "Paper copyright remains with its authors. Fetch at run time; use short excerpts with attribution.",
"retrieval": "publisher",
"bytes": 5566575,
"sha256": "82dd470712ce8389f19f20eb9330475e2166a281f8c7990a9f1d0763d73b4d22"
},
{
"slug": "fema-proof-of-loss",
"title": "FEMA Hermit's Peak/Calf Canyon Proof of Loss Form",
"publisher": "Federal Emergency Management Agency",
"file_name": "fema-proof-of-loss-form.pdf",
"url": "https://www.fema.gov/sites/default/files/documents/fema_hpcc-proof-of-loss-form-english-exp-11.30.2026.pdf",
"source_page": "https://www.fema.gov/flood-insurance/find-form/underwriting",
"rights": "U.S. federal government work. Do not imply FEMA or DHS endorsement.",
"retrieval": "publisher",
"bytes": 785324,
"sha256": "74ae94102750dcbabf784f2d9d69c2190c8fe287811c9ae9cf195e0902e05715"
}
]
}
Loading
Loading