From c2e7ceac29b36f59982613a52f09776e5226c054 Mon Sep 17 00:00:00 2001 From: svonava Date: Thu, 17 Sep 2026 09:31:43 -0700 Subject: [PATCH] docs(examples): record the FEMA bare-$ fields and which path a fetch took Two follow-ups to #280, deliberately left out of it so that PR could ship without paying another review round for a section that was already true but incomplete. Closes #288. The FEMA finding. 26 of the form's 124 non-blank lines are a bare $ and nothing else: every amount field comes back as a naked dollar sign with its label stripped, and no line in the file carries a currency amount attached to a label. It is the plainest artefact in the corpus. A table with a repeated header still reads as a table, so a reader can talk themselves out of caring; a column of 26 dollar signs cannot be read as anything but broken. And it is a form, the category where a label and its field are related only by position on the page. It also sharpens what the section already said about the checks. contains: checks have the same shape as _table_count: contains:AMOUNTS CLAIMED passes because that heading survives at line 9, and says nothing about the 26 unlabelled fields beneath it. A harness that checks facts and order will not tell you your tables are wrong, and one that checks a heading is present will not tell you the fields under it are gone. Each check answers a narrower question than the claim it is used to back. The provenance field. retrieval recorded "publisher" for both a clean first attempt and a curl fallback after a 403, so a fetch that needed the fallback read exactly like one that did not. It now records publisher-after-403-via-curl, which describes the route rather than the outcome. Why that fallback exists is worth writing down beside it, because nobody debugging a 403 would guess the direction: fema.gov refuses a request carrying a Chrome user-agent and serves plain curl, so the retry works BECAUSE it drops the browser user-agent the first attempt sends. Tested from one network only, which says nothing about what a browser on a home connection sees. No artifact changes and nothing re-run. The committed run predates the finer value and records "publisher" for all four, which is true of all four -- FEMA's arrived through the curl retry. The bytes are pinned and correct; backfilling the record would mean editing a recorded artifact to say something the run did not say. Every number checked against the recorded output rather than retyped: 26 bare $ lines of 124 non-blank, AMOUNTS CLAIMED at line 9. verify-run 65 of 65, pytest 34, ruff clean. --- examples/document-to-markdown/README.md | 52 ++++++++++++++++--- .../document_to_markdown/fetch.py | 11 +++- 2 files changed, 54 insertions(+), 9 deletions(-) diff --git a/examples/document-to-markdown/README.md b/examples/document-to-markdown/README.md index 079eee9b2..1fef199e4 100644 --- a/examples/document-to-markdown/README.md +++ b/examples/document-to-markdown/README.md @@ -159,6 +159,16 @@ rights notes say not to redistribute the complete file. `data/manifest.json` is committed instead, pinning each URL, byte length and SHA-256, which is what a reader needs to confirm they fetched the same bytes this run scored. +One of the four may refuse you. `fema.gov` answers a request carrying a Chrome +user-agent with `403` and serves plain `curl` the PDF — backwards from what +anyone debugging that 403 would guess, and the reason `fetch.py` retries with +`curl` before falling back to the bundled copy. Tested from one network only, +so it says nothing about what a browser on a home connection sees. The +`retrieval` field in `data/manifest.json` records which path a fetch took; +`publisher-after-403-via-curl` means the fallback was used. The run committed +here predates that finer-grained value and records `publisher` for all four, +which is true of every one of them — FEMA's arrived through the `curl` retry. + One call is one entry in `calls.json` rather than a file of its own. Members that nothing scores move to `payloads/` and are referenced by digest — for Docling that is `data.document`, between 85 and 98 percent of every response. @@ -222,22 +232,48 @@ multi-row cells into space-joined values — `4 16`, `177 s 167 s` — while a second table flattens an entire per-class accuracy table into two cells, one holding every label and one holding every value. +**A form loses its labels entirely.** On the FEMA proof-of-loss form, 26 of the +124 non-blank lines are a bare `$` and nothing else. Every amount field comes +back as a naked dollar sign with its label stripped, and no line in the file +carries a currency amount attached to a label: + +```text +- [ ] Other: + +$ + +$ + +$ +``` + +This is the plainest thing in the corpus. A table with a repeated header still +reads as a table, so it is possible to talk yourself out of caring; a column of +26 dollar signs is not. And it is a form — the category where the relationship +between a label and its field is purely spatial, and so the hardest for any +converter. + **Three smaller behaviours.** Fenced code blocks are flattened onto one line, so the four-statement Python example in the converted Docling paper would not run as printed. Line-break hyphens are closed up, so `MIT-licensed` returns as -`MITlicensed`. And on the FEMA form every section heading is emitted together -near the top while its fields appear much later — `TYPE OF PROOF OF LOSS` at -line 7, its six choices from line 57. The content survives; the grouping does -not. +`MITlicensed`. And on the same FEMA form every section heading is emitted +together near the top while its fields appear much later — `TYPE OF PROOF OF +LOSS` at line 7, its six choices from line 57. The content survives; the +grouping does not. **Why the checks pass anyway, and what that says about the checks.** They test exact facts, section order, and that tables are present — not that a table is the right shape. `_table_count` counts contiguous pipe blocks, so the four welded NVIDIA tables count as one, and `7 found, 4 required` passes honestly -while saying nothing about structure. That is a limitation of this evaluation, -not a detail about the model, and it is worth knowing before you copy the -approach: a conversion harness that checks facts and order will not tell you -your tables are wrong. +while saying nothing about structure. `contains:` checks have the same shape: +`contains:AMOUNTS CLAIMED` passes because that heading survives at line 9, and +says nothing about the 26 unlabelled `$` fields beneath it. + +That is a limitation of this evaluation, not a detail about the model, and it is +worth knowing before you copy the approach: a conversion harness that checks +facts and order will not tell you your tables are wrong, and one that checks a +heading is present will not tell you the fields under it are gone. Each check +answers a narrower question than the claim it is used to back. ## Honest scope diff --git a/examples/document-to-markdown/document_to_markdown/fetch.py b/examples/document-to-markdown/document_to_markdown/fetch.py index 424000671..c0b21f8ff 100644 --- a/examples/document-to-markdown/document_to_markdown/fetch.py +++ b/examples/document-to-markdown/document_to_markdown/fetch.py @@ -37,6 +37,12 @@ def _download(document: DocumentSource, destination: Path) -> tuple[bytes, str]: if error.code != 403 or curl is None: raise try: + # Deliberately WITHOUT the browser user-agent above, and that is the + # whole reason this fallback works. fema.gov serves plain curl and + # refuses the Chrome user-agent string with 403 — a WAF apparently + # reading "Chrome from a datacenter" as a scraper and "curl" as an + # honest tool. Backwards from what anyone debugging a 403 would + # guess, so it is written down rather than left to be rediscovered. response = subprocess.run( [curl, "-fsSL", "--retry", "3", document.url], check=True, @@ -44,7 +50,10 @@ def _download(document: DocumentSource, destination: Path) -> tuple[bytes, str]: timeout=180, ) payload = response.stdout - retrieval = "publisher" + # Distinguished from a clean first attempt. "publisher" was true of + # both and could not tell them apart, so a record of a fetch that + # needed the fallback read exactly like one that did not. + retrieval = "publisher-after-403-via-curl" except subprocess.CalledProcessError: if document.fixture_path is None or not document.fixture_path.exists(): raise