Repository navigation
Produce source-bound V4 assessments for the 10,000-paper index #53
Description
Why this work is needed
The authorized 10,000-paper corpus and source/native/image identities exist, but no 10,000-document V4 abstract assessment map exists. The current #22 field build therefore records zero abstract assessments and must remain a native-plus-local-OCR partial baseline. A native text file cannot be relabeled as parallel_structure_v4: that primary method consumes a genuine Docling layout document and conversion status before applying its source-bound selector.
A read-only feasibility audit of the frozen corpus found 9,975 eligible pages, 12 native extraction errors, 11 saved-page/source mismatches, and two empty-native pages (10,000 total). The private audit is validation_v3_private/issue22-assessment-map-feasibility-20260928-01/FEASIBILITY.json, SHA-256 f13fe3981810e3ba3d025e54c2fc2fb6f63cf47fc7a90f0d2cadbd5b1a13c8c9. It made no conversion/model calls and read no query inputs. These counts describe the present field policy; a producer must recheck them against its frozen exact code and source manifest.
Implementation and run contract
- Add a query-independent, unlabelled corpus producer separate from the 200-case regression runner, which requires gold labels. Freeze the final merged V4 method/import closure, local Docling/runtime versions, converter options, source-map/manifest IDs, ordering, render/native policy, resource limits, output schema and code hashes before conversion. Run a small exposed deterministic throughput/resource smoke before committing to the full workload.
- For each eligible ID, verify the authorized original, physical page-one PDF/image and selected native representation, execute real local Docling conversion, save raw conversion document/status and timing/error state, then run the frozen
parallel_structure_v4assessor. Bind every result to source/page/image/native hashes, converter document/status hashes, runtime/code identity and versioned policy. Verify inputs again before sealing. Never synthesize a failed converter output to fill a missing row. - Make the 10,000-ID execution ledger resumable and single-owner: immutable per-ID receipts, deterministic input order, explicit pending/running/complete/error/ineligible states, restart reconciliation, final count/hash readback, and no duplicate controller. Record 12 native errors, 11 page mismatches and two empty-native pages as explicit ineligible outcomes unless fresh source verification changes the policy. Do not feed those ineligible rows to the current assessment-map consumer, which correctly rejects them.
- Produce
retrieval-assessment-map-v1entries for every eligible row, including actual held/error outcomes. Verify source spans and proposal text against the exact native representation. Keep converter errors, uncertain/partial/absent states, review-only proposals, notation holds and source-reviewed corrections distinct. No unreviewed proposal qualifies graph evidence or automatic fallback. - Rebuild all 10,000 query-independent fields with that map and the independently bound local OCR policy for the two empty-native pages. Hash fields, regenerate affected dense vectors, reject stale caches, and rerun the frozen 60-query candidate/rerank protocol and target-label rotation plus ablations. Report per-stage recall, timing, resource use, missed/held IDs and all 10,000 denominators. Resolve the separate mixed metadata/title defect Preserve title lines when classification metadata shares a native block #52 before claiming the final repaired-field comparison.
Acceptance
- A real exposed-page Docling→V4 smoke measures time/memory and proves the exact converter document/status and selected source spans; the declared full-run resource budget is based on that measurement.
- Authored tests reject native-only masquerading as Docling, wrong source/page/native hashes, forged or stale converter receipts, duplicate/missing IDs, query/target-dependent field selection, and unsafe resume after interruption.
- A complete 10,000-ID ledger and eligible-only assessment map verify by independent readback. Every eligible row has a genuine converter attempt and sealed V4 outcome; every ineligible row has a specific source-policy reason. Failed attempts remain failures.
- Field and ranking receipts bind to the new map and exact method/runtime tree; old dense caches fail. Full candidate/rerank results, rotation and ablations use the same frozen corpus/query inputs and account for all missing/held cases.
- Public code, protocol, tests and redacted aggregate receipt are delivered through signed PR/CI/merge. Private PDFs, converter text, labels and per-paper rankings remain external. No paid provider calls or production graph writes occur.
Relationships
Native subissue of #22, under epic #12. Implementation and measured claims wait for #19's final combined extraction tree and source-reviewed 200-paper gate. #52 independently blocks the final repaired-field title/ranking interpretation. The existing #22 native-plus-OCR result remains an exact-hash partial baseline.
Activity
CompleteDotTech commentedon Sep 29, 2026
Producer mechanics delivered in PR #69, merged at
cd5dd893d2742ce4f3ef14d040859f825caecbdd. The signed final head passed six hosted PR checks; exact-head pinned Python 3.12 validation ran 1,155 tests (1,147 pass, 8 skip, 0 failures/errors), and package validation passed. All three post-merge main jobs also passed.The implementation performs real offline Docling conversions in an owned resident CPU worker, seals source-bound V4 assessments and per-ID receipts, records ineligible native/page states, preserves partial conversion documents under error status, and fails closed on unknown attempts. A fresh exposed exact-head smoke produced one genuine eligible conversion and one page-mismatch ineligible row (private ledger SHA-256
876e47845eee69a42c66c37ffafa800e71a093b56ec39d244663870d5cbac21d). This is mechanics evidence, not the 10,000-paper map.#53 remains OPEN. Final #19 selector and resource profile must be frozen before the full 10,000-ID run; after that, verify the eligible-only map, rebuild fields/dense cache, rerun all 60 queries and ablations, and publish the aggregate receipt. The exposed ten-page throughput estimate (~18.3 hours at its observed rate) is only planning evidence; reserve over 24 hours including tails/recovery.
CompleteDotTech commentedon Sep 29, 2026
Full-corpus run stopped; evidence preserved
The genuine local Docling→V4 full-corpus producer stopped with
abstract_field_spans_not_unique_or_currentat the 109th frozen ID, after 108 sealed per-ID assessment/row receipts. The next genuine converter attempt completed successfully and its intent, conversion document and result are preserved. The final ledger/map do not exist, so this is not a completed 10,000-paper assessment run.Confirmed cause: the retrieval verifier relocates the entire multi-paragraph transcription as one contiguous native sequence. On the failing two-column page, native extraction interleaves right-column Introduction lines between three left-column abstract paragraphs. The selected left-column spans are exact; each paragraph separately relocates uniquely with positions identical to the saved spans. The full concatenated text cannot satisfy the single contiguous-chain assumption. Source review confirms the abstract's left-column ownership and closure before classification metadata.
Repair contract: verify exact source-bound paragraph groups using their located region/section evidence, unique relocation, complete saved-position correspondence, column ownership, reading order and closing boundary. Retain negative checks for arbitrary omissions, invented group breaks, cross-column contamination, nonunique matches and stale source spans; do not lower the alignment threshold.
Recovery contract: the frozen protocol pins code identity, so changing code and resuming under that protocol would invalidate provenance. Preserve the stopped run untouched. A separately versioned migration must verify all 108 existing receipts plus the pending intent/conversion/document, pin their hashes, reassess under the repaired verifier and explicitly record reused conversions in a new output identity. No conversion is relabeled as newly executed. Resume only after code review, meaningful tests, delivered identity verification and ownership reconciliation.
The source-gate delivery in #19 is being completed separately. This failure supplies no unseen qualification, full retrieval acceptance or graph-admission authority. No shared workloads were stopped; paid API calls and graph writes remain zero.
CompleteDotTech commentedon Sep 29, 2026
Recovery implementation is published in PR #84 at verified signed head
3ec35439. Independent code review and required pinned validation pass: 1,284 tests run, 1,276 passed, 8 skipped, zero failures/errors. Source ZIP audit and isolated wheel installation pass. All six hosted jobs are running; merge remains pending their results.A fresh read-only audit replayed all 108 completed original rows and sealed a migration plan for 109 genuine conversion attempts. Plan SHA-256:
84a277f1f3172bdb7e1904238af5a01cef00f3687ff680b3d0f4ae42ad0bc88b; no new model call occurred. The frozen old run remains untouched, and full migration is NOT RUN.A fresh 200-paper/four-arm development regression is running against the new measured closure. Source/image acceptance and delivered verification must pass before the migration controller starts. This does not establish a complete assessment map, final retrieval result, unseen qualification or generated-answer pilot.
CompleteDotTech commentedon Sep 29, 2026
PR #84 is merged as verified commit
5e806197e62a2bdd94be13df4a8e321ce3a091f7. All six reviewed-head checks and all three default-branch checks pass. The delivered tree is byte-identical to signed reviewed head3ec35439; a fresh delivered-source process passes all five paragraph-group smoke tests. Required local suite/package/wheel evidence remains linked in the PR.The recovery code is delivered. The full 10,000-document migration remains NOT RUN while the fresh 200-paper/image-enabled acceptance finishes. This issue remains open for the complete ledger/map and retrieval evidence; implementation delivery alone does not satisfy those empirical criteria. The old stopped run and its 109 genuine conversion attempts remain intact.
CompleteDotTech commentedon Sep 30, 2026
Fresh source acceptance and guarded corpus recovery
After delivery of #84, the actual current-code 200-paper regression and image evidence have been refreshed and independently replayed against delivered
main(5e806197e62a2bdd94be13df4a8e321ce3a091f7).91cd57cffdc32fb0b7e5736ae62da398d3065e07a373095247a65f4024cdc8b6; policy SHA256c5da4e36447249acc7cf19eee2ad236ba693fd203780f1f1fa2e6a158fb6e1d3. The lead's actual delivered-main verify exited 0/PASS; verify output SHA25647e7e1a0a504e0d8b7e21f0b8dfdb5d686cb42b46a8df3ecc76d9999a1efe734.The sole guarded migration supervisor started at 2026-09-30 00:06:41 UTC, after review of its exact controller and launch pins. It requires three consecutive host-memory samples at or above 6 GiB and a fresh prelaunch sample, preserves the original resource window, and holds the original supervisor lock. Old controllers/workers are absent; the frozen origin run remains unchanged.
Fresh useful progress is verified: the new protocol is sealed and 21 imported rows were present in the approximately 00:08 UTC readback, with sampled source/conversion/assessment hashes verified and
new_model_call=false. The immutable migration plan authenticates 109 sealed origin conversion attempts; genuine new CPU Docling conversion follows the import phase. No duplicate producer was started.This is recovery progress, not full-corpus qualification. The full 10,000-document map, rebuilt final retrieval fields/ranks, unseen qualification and real paper-to-answer pilot remain outstanding. #53 stays open. Related consumer repair #85 has passed local release gates and awaits exact-head hosted delivery checks; qualification controller pins will be refreshed after its merge.
CompleteDotTech commentedon Sep 30, 2026
Full 10,000-document producer status
The owned CPU Docling run is active and has progressed beyond its 109 verified migrated rows. At the 01:05:46 UTC readback on September 30, 557 rows were sealed; the producer had no stderr output. This is an in-progress count, not a final assessment or retrieval result.
The single resident converter's thread count and private memory commitment have grown substantially during sequential pages. Per-document input/output medians have remained broadly similar, but the evidence does not yet identify which Docling, OCR, PyTorch, or native allocator retains resources. The run remains live and has not been restarted or modified.
A follow-up implementation is being developed in a separate checkout to bound the number of documents processed by each owned converter child. At a fully verified row boundary, the controller can send its existing graceful stop command, wait for child exit, seal a session-end receipt, and start a new child with a new session-start receipt. That change needs an isolated code version, durability tests, and a migration/reconciliation path that preserves existing receipts and treats an unfinished attempt as unknown rather than silently repeating a model call. It must not be patched into the active frozen run.
The current run will continue under resource and row-progress observation. Final success remains contingent on complete source-bound assessments, the full-run ledger, and the existing acceptance gates.
Private investigation receipt SHA-256:
13b6c5273b510a08e4b02328095ffe875a0d158905642ba13e57dfbc500738c2. The observed thread and memory counts include decreases as well as increases, so a permanently leaked per-page thread count is not established.Research context: Docling's pipeline source creates per-run stages and includes shutdown/join handling; the installed 2.130 source was inspected directly. PyTorch's multithreaded inference report describes a similar class of memory growth. These references support investigation and bounded process residence; they do not establish the upstream cause of this worker's retained threads.
CompleteDotTech commentedon Sep 30, 2026
Bounded converter implementation ready for hosted checks
PR #86 contains the independently reviewed lifecycle repair. Signed head
0dc2863eda2a54a73c132ce2161c61f1d8e15d33is verified by GitHub. Local validation ran 1,303 tests: 1,295 passed, 8 skipped, zero failures/errors; 25 focused lifecycle tests and the harmless actual Windows launcher/worker smoke passed. The source ZIP and installed core-library CLI smoke also passed. Six hosted checks are currently in progress; merge is pending.The repair bounds future sessions, binds timeout cleanup to the verified actual worker rather than the launcher, and exposes a trusted local drain handshake with a distinct partial-run exit. It does not prove an upstream permanent memory/thread leak or claim a genuine Docling rotation experiment has already run.
The existing frozen 10,000-document producer remains live and unchanged, with 858 sealed rows at the last verified checkpoint. Its run identity, source receipts, original planning window, and unknown-outcome holds are preserved. Full map and retrieval acceptance remain outstanding; this issue stays open.
CompleteDotTech commentedon Sep 30, 2026
Converter lifecycle fix delivered
PR #86 is merged as
4ab4fb5e6e8ca684478a874b2699fab8e1228063. All six checks on final head7acc1865756a1536b72a1b726acc1c6f2b18b3f3and all three main-branch checks passed. Final local validation ran 1,303 tests: 1,295 passed, 8 skipped, zero failures/errors; 25 post-merge lifecycle tests passed. The merge tree exactly matches the tested head. Both commits and the merge are signature-verified; unrelated local work and the frozen live producer were preserved.The initial Linux CI errors were reproduced and fixed in test isolation: mocking global os.name altered pathlib semantics, and one Windows handle test omitted platform simulation. The actual implementation remained unchanged by that follow-up; all 25 focused Windows tests and 23 Linux tests plus two expected skips passed.
Delivery readback SHA-256:
24b8e2c3e0ee34d48963a6226ac306cf122ee0a16612bd188eea794163087f04. Source ZIP SHA-256:bff7b7055102a2b2d99b412774c583e67ea69684efe824832e4d545e87f93374.The long-running corpus producer remains on its original frozen code, with its identity and receipts preserved. Genuine Docling bounded-session/drain smoke and complete 10,000-document assessment/retrieval acceptance remain outstanding. This delivery does not close the empirical gate. The broader extraction code-identity closures changed, so a fresh current-code 200-paper/image receipt is required before prospective qualification; the historical passing source-verified result and failing legacy scores remain unchanged.
CompleteDotTech commentedon Sep 30, 2026
Stopped corpus run: checkpoint preserved; bounded recovery in progress
The frozen corpus producer stopped unexpectedly at 2026-09-30 02:22 UTC with exit 1. Readback found 1,207 contiguous sealed rows, an empty next-attempt directory, and no later attempts. The independent current-code 200-paper regression stopped during paper 29 with an
OSError; its partial output is also preserved.Cause is unresolved. The corpus traceback ends before the exception line. Examined Windows event channels showed no corresponding resource-exhaustion, Defender, application, or storage failure. Recomputing the affected cached 200-paper arm in a new diagnostic output passed. Earlier resident-worker thread/handle growth is a plausible resource-pressure mechanism, not a proven cause of this incident.
The frozen controller publishes the immutable intent before requesting conversion. The empty next-attempt directory therefore supports a pre-call interruption; no unknown conversion is being retried. Existing source/row/converter receipts remain unchanged.
Recovery gap and repair
PR #86 delivered bounded sessions and graceful drain, but the existing migration API cannot import this interrupted checkpoint: it requires a sealed pending conversion and assumes each origin row has a local attempt. This checkpoint mixes 109 historical imports with subsequent genuine conversions, followed by an empty pre-intent directory.
A versioned mixed-prefix migration is being implemented and independently reviewed. It must replay every sealed origin row and its converter lineage, reassess cached converter documents under the delivered code without new model calls, preserve the original output, and start only the remaining conversions in a new bounded-session output. Unknown outcomes must hold. The old unbounded worker remains stopped.
The fresh 200-paper retry has a reviewed wrapper that retains full private traceback/OS error details. It has not launched because the unchanged 10 GiB memory guard has not passed. Current-source qualification, the complete 10,000-paper map, and downstream retrieval acceptance remain pending.
Private reconciliation checkpoint SHA-256:
97c3ecb96181d189eca82de12feced210bac6f54fee4547b26747be946a675ce.CompleteDotTech commentedon Sep 30, 2026
Reopened: the complete 10,000-ID ledger, eligible assessment map, and field/ranking acceptance criteria are still pending. At PR #86 merge, GitHub interpreted the closing keyword immediately before this issue reference inside a negated sentence; that wording has been corrected. PR #87 provides the next reviewed recovery repair. The underlying host failure remains unresolved and the original stopped-run receipts are preserved.
CompleteDotTech commentedon Sep 30, 2026
PR #87 merged as
31268c8b6fb03fa06b811dfecf876052590df2dc. Reviewed signed head0b5db8ea4dc304d2a4c4b37e2fed464693c41b31passed all six hosted Python checks and the exact-head full suite: 1,311 run, 1,303 passed, eight explicit skips, zero failures/errors. Merge tree equals tested tree; post-merge migration tests passed 8/8. Main-branch CI is running.The mixed-prefix recovery plan preserves and source-replays all 1,207 sealed documents, the empty next pre-intent directory, and the immutable origin. All 83 planned current-method files match the delivered code. Preparation imported no new model observations and has not started fresh conversions. New current-source 200-paper qualification and the bounded 10,000-document recovery remain pending.
Issue #53 remains open for the complete assessment map and downstream field/retrieval acceptance. A previous negated closing phrase in PR #86 unintentionally triggered GitHub automatic closure; its body has been corrected and #53 reopened, preserving the event history. Legacy scores and the separate source-verified gate remain unchanged.