Repository navigation
Qualify the frozen extraction configuration on an unseen internally reviewed paper cohort #20
Description
Why this is required
The 200-paper corpus has been inspected and used to diagnose selector changes. It is a development regression set. The eight newer papers were reviewed by an assistant and assessed through a native-only diagnostic: 1/6 complete abstracts was correctly proposed, with two additional pages marked uncertain for notation. Neither set establishes held-out full-pipeline quality.
Freeze the improved configuration after the integrated regression gate and evaluate a genuinely unseen cohort with references produced through blinded internal source-first review before model predictions. Report the reviewer kind accurately; this owner-approved protocol does not claim independent human validation.
Starting points
- Source preparation.
- Exposure checks and reference freeze.
- Freeze verification and cohort execution.
- Current source-first protocol and known limitations.
The existing freeze records review attribution but does not certify human independence. Preserve attribution, blinding and source evidence; an unseen_source_reviewed string alone proves none of these.
Protocol to finalize before opening predictions
- Register the evaluation question, sampling frame/seed, sample-size justification, numeric precision/recall/fidelity targets, confidence-interval method and stop/reject rules. Specify representation of complete/partial/absent abstracts, multi-column/structured/title-page layouts, notation and image-only/broken text sources. Targets must be set before results; this issue does not invent achieved quality.
- Freeze selector, converter/model, runtime, metric and fallback configuration. Include the verified image-evidence route in the final qualification configuration and complete its implementation before this freeze. Native-only intermediate experiments must be labeled with their narrower scope.
- Build an exposure registry covering all earlier regression, retrieval, diagnostic, source-review and development samples, deduplicating by work identity and source/page hashes. The known 200 registry alone does not cover all prior exposure.
- Have an attributed internal reviewer or review agent inspect original source images and spans before any candidate predictions/conversion outputs are examined. Record reviewer kind, identity/role, review timestamps, scope and blinding evidence; use specialist notation review where available. Preserve disagreements and uncertainty with an adjudication protocol. Assistant review must stay labeled assistant review and must not be presented as independent human review.
- Hash-bind labels, source/page/image/native evidence, exact spans or typed image transcription, boundaries, protocol and reviewer receipts. Keep all private artifacts under the authorized external root.
- Generate authentic converter outputs only after the reference/code freeze, run all selected arms, and report per-stratum plus aggregate error/coverage metrics and confidence intervals. Record conversion/truncation failures, unresolved references, latency/cost and review burden; do not drop difficult cases.
- If any method is tuned after seeing results, designate that cohort as exposed, create a new unseen cohort and repeat. Failed qualification remains a failure with its evidence preserved.
Executable starting runbook
Create external sources.json containing sample_id, work_id and source_relative. Choose a new directory before running:
export TRACE_GC_TEST_DATA_ROOT=/authorized/TRACE-GC_RealPaper_Test
python -m src.parallel_source_v4.prepare \
--data-root "$TRACE_GC_TEST_DATA_ROOT" \
--sources validation_v3_private/qualification-UNIQUE/sources.json \
--output validation_v3_private/qualification-UNIQUE/source-viewsComplete blinded internal source review and record its evidence before:
python -m src.parallel_source_v4.freeze \
--data-root "$TRACE_GC_TEST_DATA_ROOT" \
--manifest validation_v3_private/qualification-UNIQUE/source-views/manifest.json \
--labels validation_v3_private/qualification-UNIQUE/labels.json \
--exposure-registry validation_v3_private/qualification-UNIQUE/exposure-registry.json \
--output validation_v3_private/qualification-UNIQUE/freeze.jsonAfter producing authentic authorized converter outputs at the manifest paths:
python -m src.parallel_source_v4.extraction cohort \
--data-root "$TRACE_GC_TEST_DATA_ROOT" \
--freeze validation_v3_private/qualification-UNIQUE/freeze.json \
--output validation_v3_private/qualification-UNIQUE/results-runThese are existing harness commands, not a claim that they already enforce every new independence or fidelity requirement. Extend them where the protocol requires it.
Acceptance criteria
- Integrated 200-paper gate passes for the exact candidate configuration, including a fresh integrated receipt if adding the image-evidence route changes the earlier measured configuration. The image-evidence implementation is complete.
- Registered sampling/targets/methods are timestamped and immutable before predictions; all earlier exposures are excluded by identity, not filename alone.
- Blinded internal source-first review and notation adjudication actually occur, with inspectable attribution and source-bound evidence. Report reviewer kind and unresolved cases; no independent-human claim is made.
- Frozen references, code, environment and configuration verify through a completed run with all selected genuine converters.
- The measured result meets every preregistered acceptance target with the specified uncertainty calculation. A failed run is documented and does not close this qualification issue as successful.
- Redacted aggregate report records sample counts, strata, denominators, failures, unresolved reviews, confidence intervals, latency and actual spend; references/full assessments remain private.
- Post-result tuning triggers a new evaluation cohort, and the promotion receipt names this qualification evidence plus the regression evidence.
- No first-page quality result is presented as validation of later-page ingestion, graph publication or generated answers.
Research and operational scope
Selective Classification motivates reporting coverage together with accepted error rather than rewarding near-total abstention. OmniDocBench motivates separating layout/content attributes.
Corpus/internal review capacity and authorized converter availability are execution prerequisites. Existing local cached tools may be used within established authorization; this issue does not grant paid-provider spending or permission to publish private papers. Missing inputs must remain explicit rather than being replaced with synthetic data under a held-out-quality claim.
Tracking and completion dependencies
Parent: #12. Milestone: Paper extraction and GraphRAG qualification.
Completion prerequisites: #19, #18. These are native blocked-by links; design, fixtures and other independent implementation work may proceed in parallel.
Required by: #23.
Activity
[-]Qualify the frozen extraction configuration on an unseen independently reviewed paper cohort[/-][+]Qualify the frozen extraction configuration on an unseen internally reviewed paper cohort[/+]CompleteDotTech commentedon Sep 28, 2026
Preregistration checkpoint (2026-09-28): the attributed internal source-first protocol and metadata-only sampling frame are frozen. Current preregistration receipt SHA-256:
0e004bf9990653d3b44dc2d61291ad4acb02d33266eb21464de5452ce7879b0c. The exclusion registry covers 156,886 previously exposed works. Official arXiv OAI metadata produced two disjoint 2,000-work candidate frames (recent and historical), with zero overlap against that registry and zero cross-frame overlap. The initial export API HTTP 406 attempts and documented OAI no-record result remain in the private audit; the recovery used the official metadata service.No candidate PDF, image, converter output, or prediction has been opened for this cohort. The protocol remains blocked at
FRAME_READY_CONFIGURATION_BLOCKEDuntil an exact image-enabled #19 PASS/200 receipt, authentic converter access, and the qualification harness checks exist. The preregistered primary sample is 400; the separate source-only challenge screens are 600 + 600. Targets were frozen before any paper content was inspected. This is a protocol checkpoint, not a qualification result. External independent human review was waived by the owner; any eventual review will be attributed internal review, without an independence claim.CompleteDotTech commentedon Sep 28, 2026
Implementation update: PR #42 merged at 68a038b (GitHub verified signature). Six PR checks passed on 41ec89b, and all three Python 3.11/3.12/3.13 postmerge main checks passed in run 36479603957. Integrated local validation passed 906 of 911 tests with five skips; package validation passed.
The evaluator now enforces the frozen unseen-paper protocol, exact bounds, zero false accepts, replayable publication and invalidation, pinned source/runtime inputs, and attributed internal source-first review. No unseen 400-paper sample, 1,200 challenge screens, or real predictions have been run. This issue remains open until those empirical acceptance gates pass. Per owner scope change, no independent human review is claimed.
CompleteDotTech commentedon Sep 29, 2026
Execution-gate update after merged PR #50: the exact image-enabled reviewed 200-paper configuration passed its private acceptance and full verifier on measured extraction commit
7cbc22fb217f2ccac822f7e3badf2815fdb3ea13(acceptance SHA-25640a6263d5c7b25100ba4a0b8bf46cb36459e93320446b87c5dc6afb5b1412477; verifier SHA-256699d826d7cfd5f0cc99b71b7bfb145f8be4ff2030daf934254dcb09ab001347c). PR #50 is docs/receipt only, signed and merged with six PR plus three post-merge checks PASS. This discharges the empirical development gate for that exact measured code/configuration; it is not unseen qualification. A newly reproduced #17 metadata-suffix safety defect is being repaired; any changed extraction code requires a new 200-paper receipt before the prospective freeze.The preregistered metadata-only frame/exposure inventory remains frozen (400 recent primary and 1,200 challenge-screening positions; no prospective PDFs/images/native text or predictions opened). A read-only local converter inventory shows no active Docling/GROBID/MinerU/olmOCR service or those packages in the current shell/validation Python environments. A project GROBID 0.9.1-full container is stopped with exit 137; Docker does not mark it OOM-killed, so its cause is unproven. The local olmOCR snapshot visibly has only 2 of 4 model weight shards. This is a setup/access inventory, not proof that all converter assets are unavailable elsewhere. Genuine four-arm availability remains unverified, and no outputs will be simulated to bypass it.
Next gate: finish #17 and remeasure #19 on the final code tree, then bind actual converter/runtime assets and attribution before opening selected unseen source pages. Internal assistant source review follows the owner-approved waiver; no independent-human claim.
1 remaining item
CompleteDotTech commentedon Sep 29, 2026
Container configuration readback: the prior GROBID container ID is
f7944ca82da3416d0cd8fdbe5f727f572db57fd9539c6375e7fe393b1457dd5e; it pins 4 GiB memory, 4 GiB memory+swap, and 2 CPUs, with command./grobid-service/bin/grobid-service. These are historical settings, not proof that the service will meet the new smoke budget. The Docker executable is installed locally; the exact executable/JAR/model/config file hashes and live service response still need a fresh pin. No restart or model call was made.CompleteDotTech commentedon Sep 29, 2026
Local converter availability update (2026-09-29): a complete-looking olmOCR 2-7B-1025 snapshot is present under the authorized external data root at
validation_v2/models/allenai--olmOCR-2-7B-1025(four safetensors shards, index, config and tokenizer files; shard bytes 4,968,243,272 + 4,991,495,784 + 4,932,751,008 + 1,691,924,352). A local Qwen3-4B-Instruct-2507 snapshot is also present undervalidation_v3_private/issue23-qwen3-4b-instruct-2507-cdbee75. These are availability findings, not a successful model load or unseen-paper evaluation. The held-out protocol, integrated regression and resource-safe converter execution remain required; no paid inference or private-PDF transfer has been authorized.CompleteDotTech commentedon Sep 29, 2026
Local converter-resource readback for the upcoming unseen-cohort gate: a complete
allenai/olmOCR-2-7B-1025BF16 snapshot is present outside the public repo. Its four safetensors files total 16,584,414,416 bytes (15.45 GiB). The local RTX 3060 has 12,288 MiB total VRAM (9,619 MiB free at this check), so the BF16 weights alone cannot fit in GPU memory. No unseen-paper converter run or FP8/4-bit quality claim has been made from this discovery.Concrete local next step: validate an exact, hash-bound lower-memory model/runtime configuration on an already exposed diagnostic paper, recording output fidelity, truncation, latency and peak VRAM. Only after that smoke passes should the blinded source-first unseen cohort and converter configuration be frozen and run. Keep the BF16 snapshot and private PDFs external; paid remote inference or PDF transfer has not been authorized. The owner has allowed attributed internal source review, so reviewer independence must not be claimed.
CompleteDotTech commentedon Sep 29, 2026
Correction/clarification to my prior local-model comment: the 15.45-GiB figure is the BF16 on-disk weight size, not the resident size under the repository's existing NF4 4-bit load path. The private converter inventory (
validation_v3_private/issue20-local-converter-readiness-20260928-01/SMOKE_PLAN.md) documents prior genuine local olmOCR conversions on this 12-GiB RTX 3060 using NF4 double quantization with BF16 compute. A remote endpoint or new FP8 snapshot is therefore not inherently required for the exposed-page smoke. That historical execution does not qualify the current frozen code/runtime or an unseen cohort.The current source-ready invocation readback is
validation_v3_private/issue20-converter-smoke-invocation-20260929-01/PLANNING_READBACK.json:WAIT_FINAL_CODE_AND_RESOURCE_WINDOW, zero new converter calls. At the latest host check, free RAM was about 15.8 GiB, below the private smoke plan's conservative 20-GiB threshold, while shared workloads remain active. Do not evict them to make room. The next local action is one-at-a-time f097 exposed-page execution with fresh source/model/runtime/code hashes and actual resource measurement after the final selector tree and a safe resource window; preserve failed/truncated attempts as such.CompleteDotTech commentedon Sep 29, 2026
Prospective-source audit (read-only, 2026-09-29): the existing private preregistration has two hash-bound arXiv OAI-PMH metadata frames, 2,000 recent and 2,000 historical challenge works. Their work IDs have zero known overlap with the conservative local exposure registry (156,886 observed works / 191,454 catalog versions). That registry excludes the available local PDFs as unseen candidates; verified eligible local PDFs: 0. The 4,000 metadata candidates are not yet an eligible PDF cohort: no selected versioned PDFs have been fetched/opened, and PDF rights, aliases, bytes, page hashes and source reviews remain unverified.
The preregistered sequence requires exact image-enabled configuration, current #19 PASS, converter availability and a frozen identity/alias screen before selected source pages are opened. None of those conditions is being bypassed. Attributed blinded internal assistant review is allowed by the user's waiver; no independent-human claim is made. No prospective PDF, model call, paid inference or private upload occurred in this audit. #20 remains OPEN.
CompleteDotTech commentedon Sep 29, 2026
Local converter readiness update (2026-09-29): a private local snapshot of
allenai/olmOCR-2-7B-1025at revisione52d6f090b7a9007afffbbd6ce510876222fea93is present. Its index names 729 tensors across four safetensors shards; all four files exist and their headers expose all 729 indexed tensor keys, with no missing or extra keys. A historical local log records successful page inference from this snapshot. This establishes local model availability, not a currently running endpoint or final selected-configuration accuracy.Current shared RTX 3060 capacity is constrained (12 GB total, about 1.7 GB free at the audit); no new model job or competing workload was started. Earlier local olmOCR results used NF4/direct Transformers decoding and explicitly differed from the upstream BF16/FP8 pipeline. Exact selected converter/configuration qualification, #19 PASS, eligible prospective PDFs, and source review remain outstanding. #20 stays OPEN.
CompleteDotTech commentedon Sep 29, 2026
Issue #20 local unseen-pool inventory (metadata only)
A local versioned arXiv PDF collection contains 191,454 PDF paths representing 156,886 unique work IDs. A conservative union of 144 private cohort, registry, manifest, source-map, protocol, and selection metadata files marks 30,320 of those works as exposed or present in a prior sampling frame. This leaves 126,566 provisionally unexposed works represented by 148,986 local PDF paths.
This is a source-availability inventory, not a frozen qualification cohort. The audit read directory entries and metadata JSON only. It did not open or hash PDF bytes, inspect predictions, run inference, or choose a sample. Before qualification, reconcile any exposure records outside the scanned metadata, verify source hashes and aliases for sampled files, preregister the source-first sampling frame and targets, then freeze the exact system configuration. The existing prior unseen-study PDFs and the 10,000-document retrieval corpus remain excluded.
Private receipt:
validation_v3_private/issue20-local-pool-audit-20260929-03.json, SHA-256fba049a019a4f2a612ca57050e50631fd8fb858078580b2d59ea83e225017874.CompleteDotTech commentedon Sep 30, 2026
Delivered prospective consumer repair
PR #85 is merged as verified commit
8d89c832cd856166ce000fc64344ba720c4bae6f. All six exact-head checks and all three delivered-main checks passed. The pinned evaluator reran 61 consumer tests on main: PASS. Local source packaging, no-deps wheel installation and full release validation passed (1,284 passed, 8 skipped, zero failures).The consumer now replays source-bound alias exclusions without changing deterministic ranking, verifies title-page EOF/page-two evidence, requires blinded page-two source access, and checks reference extent even when the selector abstains. The scientific continuation judgment remains an attributed reviewer decision. Numeric targets, 400 primary / 1,200 screening samples, eight quotas and four genuine converter arms remain unchanged.
The fresh integrated development prerequisite is satisfied under the owner-approved source gate: all original 91 IDs preserved, 97 source-verified proposals, all 24 current reviewed image routes valid. Receipt SHA256
91cd57cffdc32fb0b7e5736ae62da398d3065e07a373095247a65f4024cdc8b6; unchanged legacy f081 encoding failure remains visible. The delivered source identity mapping confirms unchanged measured V4/image/source-gate code.Four f097 exposed-development readiness drafts are refreshed against delivered code; they remain DRAFT pending actual asset/readiness checks and genuine engine probes. No prospective cohort or predictions have been opened. Known work/source/page/observed DOI aliases must be reconciled before freeze. A single public metadata probe returned HTTP 429 and stopped without retry; missing publisher DOI coverage is explicitly unknown, not a claim of DOI absence and not a new invented acceptance gate.
#20 stays open for blinded internal reference review, actual four-arm execution and every preregistered empirical acceptance criterion. This implementation delivery is not unseen qualification.
CompleteDotTech commentedon Sep 30, 2026
Fresh 200-paper regression completed on delivered main
d908283d94d44c22b8427b5ad463493424c5c967, using fresh native extraction andsource_fraction_v1geometry with all four cached converter arms. All 200 native files and 800 assessment files were written; root verified every assessment digest against the sealed result.Saved legacy replay passed. The unchanged legacy 91-ID preservation gate failed: f079/f122 are withheld for source-bound leading-prefix evidence; f081/f153 for verified column continuation. Each candidate passes text98, but that does not establish source fidelity or publication authority. Process exit was 1; result status
FAILED_REGRESSION_PRESERVATION_GATE.Result SHA256:
4bf0f2b3241049a438ce45a408c0bcfe43d18f3da67586a43120b6aaba5f5712. Wrapper receipt SHA256:ce36804a341e3cb957e8e1b80d6e184f97c9c7b968f4603dd1e5af786a87a2b5.This fresh native/cached-arm run does not supersede the separately accepted source-verified 91-ID gate. Next: bind the reviewed image/source evidence into a new integrated receipt for the delivered configuration, preserving these legacy results. The unseen cohort remains unfrozen/unqualified pending that integration; no independent-human review is claimed. Paid API calls and production graph writes were zero.
CompleteDotTech commentedon Sep 30, 2026
Current-runtime image-route recovery: completed replay, integrated gate pending
The fresh 200-paper run reported four withheld original-success papers (f079, f081, f122, f153). All 24 source-reviewed image routes have now been replayed through the actual current-runtime image selector, including those four papers. Each route passed its source-bound fidelity check; graph admission remains disabled. This is previously exposed development evidence with attributed assistant review, and does not qualify the unseen cohort.
The first replay stopped because four following-page closure assets were absent from the relocated test-data mirror. The assets were copied with source/destination SHA-256 verification. A versioned resume script passed an all-24 actual API preflight and separate code review before execution. It completed with exit 0 and preserved all 32 preexisting output files unchanged. The failed attempt remains recorded.
Immutable receipts:
fca014a79728e3e561bd01b6bafb57a40baf63d08fe649bcbbb39b7e4413691f.9d22efc49b57ab18c00f0bdc221142c9f29c695c6459ef87bd750dfc5602e6f0.faaaac2b3415e5e391317832d363b5f28ec6c74a9a74fa625ce008639236fbe5.91da249badff9afb984a96890163baaf9efe06d7fdddda35d9f4b61355729edf.The integrated source-verified selector gate has not yet been assessed for these current artifacts. The unseen sample/configuration freeze remains pending that gate; issue #20 remains open. Historical reference attribution and legacy scores are preserved. No new paid-provider calls or production graph writes were made.
CompleteDotTech commentedon Sep 30, 2026
Current delivered-runtime integrated source gate: PASS, replay verified
The current delivered main runtime (
d908283d94d44c22b8427b5ad463493424c5c967) completed the integrated 200-paper source-verified gate and a separate fresh receipt replay, both exit 0. All 83 method files and 124 image-code files were rehashed against the receipt and match the delivered runtime.Immutable receipts:
3b1406c2036f8f9795f5845e910d99aef5688997a627b28d4f454345d8637063.44895d6c022f0d6cd68ea8cccd5dcffade5d3995a31f2f5cce49a607460867f9.63ca4b1a0b1e043fa1bd0c15c6d7e89d8b37a6995773ac8c3fa6c7f334db8781.c7d895fda1a134b8375017e4c7198a29dfd02bbda17852cc351fb782816d93bc.This satisfies the current integrated development-regression prerequisite. It is attributed assistant review of an exposed corpus, not unseen qualification or independent human validation. The unseen sample/configuration protocol, genuine converter execution and preregistered targets remain outstanding, so #20 stays open. No graph admission, paid-provider calls or production writes were authorized by this receipt.
CompleteDotTech commentedon Sep 30, 2026
Current delivered configuration: converter readiness progress
The source-verified 200-paper prerequisite remains PASS. Additional current-code readiness checks now establish:
1bb2ce50d2148087c17046b537cdb91728be014725d65db45a725f7797347607.validate_plancalls passed for Docling, GROBID, MinerU and olmOCR, including the exposed development page, profiles and declared inputs. Receipt:f315855a474d268a0cb84c6b0344aadfe4da662e98aa92c2d42c7b2cd405596a.trace-gc==0.4.0distribution and 180 repository files. Missing packages and changed pinned versions were both zero. That failed receipt is preserved:4938543ccfb64fd21fb2d4feed7e7227cacdb20ef808d8dc657a8681fcb93e00.16c8d92b46b606e1acee0282eb5f37936bd53d164468a13522dad39aaf6d9b27. Future workers must use that clean source configuration; inventory requirements remain intact.These checks loaded no model, started no service and executed no converter smoke. The four plans remain drafts; real engine smokes, resource/service readiness and final preregistration are outstanding.
The metadata-only exposure audit found complete work-ID coverage for the known retrieval/holdout/distractor sets and no frame work-ID collisions. DOI alias exclusion is still unverified because the exposure snapshot has no recorded DOIs. Verified alias enrichment and source/page byte alias checks remain gates before cohort/reference/prediction access. No unseen cohort has been selected. This issue remains open.
CompleteDotTech commentedon Oct 1, 2026
Exposure reconciliation: canonical identity repair verified
A read-only review of additional exposure metadata found a case-sensitive join defect in the new private reconciliation script. Those manifests use
arXiv:while the frozen exposure registry uses canonical lowercasearxiv:identities. Raw comparison reported zero overlap even though canonical comparison showed the works were already exposed. The existing frozen selector's normalizer was not the defective component.The repaired successor applies the same validated, versionless work-ID normalization as the frozen registry, retains raw and canonical identities with their origins, and reports preparation manifests separately from the exposure registry. Historical outputs remain preserved and are superseded for eligibility use.
Actual results from the 30 captured and rechecked metadata inputs:
Evidence identities:
3c3edbf9e63729b91edbd71073f51b68c14f18e12e81d8929970996492010c36.6e7d62ea092cb7b18309903c51c20a08827129f13cdd4b68c3311d0cba4e11b0.fd9b6e846efa605546409fc8e74e927c41914ca1da9f1c8a096358e517b63d98.This proves the scoped metadata repair, not complete exposure coverage or unseen qualification. The frozen database remained unchanged. No PDF, source text, prediction, selector, converter or model execution was performed for this check. Cohort selection and all-known exposure coverage remain unqualified; #20 remains open.