Repository navigation
Rerun the full 10,000-paper retrieval trial with bound fields and candidate-stage metrics #22
Description
Objective and established baseline
Measure whether repaired, query-independent fields and the existing V4 full-page lexical channel improve real retrieval, then diagnose remaining misses before reranking.
The expanded-200 report records 60 frozen known-item queries against a 10,000-paper index. Its common top-10 misses are f076 (empty native fields on an image-only cover) and f115 (a graphical-abstract cover). Historically f115 ranked 44 in full-page BM25 but 226 in BM25F and never reached the top-50 BM25F/SPECTER2 candidate union. V4 now has boilerplate-title filtering and a full-page candidate channel; the September 28 probe confirmed the corrected f115 title, not improved rankings.
evaluate already records each channel's top-k and the union, binds dense caches to field/query hashes and supports a pinned local cross-encoder. The integration report explicitly says this retriever has no measured 10,000-paper result. This issue completes that measurement; it must not close on unit tests or a 200-document subset.
Protocol and work
- Obtain the original 10,000-document identities, authorized source manifests and exact 60-query/target set. Freeze corpus/query hashes, field policy, channel depths, model revisions, resource budget and metrics before evaluating.
- Rebuild fields for all 10,000 documents using the field integration fix. Account for absent assessments, genuine empty pages, OCR/review availability and failed sources; never silently shrink the corpus.
- Regenerate affected SPECTER2 embeddings from those fields. Record exact base/adapter/tokenizer revisions, input template, truncation policy, file hashes and document/query identities. Extend reproducible tooling if the required cache producer is missing; an invented or target-derived ranking list is not a dense retrieval run.
- Evaluate fixed-depth candidate recall for field-weighted BM25, SPECTER2, full-page BM25 and their union before reranking. Include ablations that isolate corrected fields and the added candidate channel.
- Rerank only the frozen union with a licensed, authorized local model snapshot whose full file manifest and revision pass the existing loader. Report top-1, top-10, MRR, per-query target rank/miss reason, candidate/index/rerank time, memory and measured cost. Candidate union order without a scorer must remain labeled “not reranked.”
- Have attributed reviewers judge competing relevant results for misses/near misses. Keep known-item retrieval scores separate from broader relevance judgments. Freeze a later independent query set if tuning occurs on these 60 queries.
A reranker cannot retrieve a document excluded from its input pool; Sentence Transformers' retrieve/rerank documentation motivates explicit stage measurements.
Existing entry point
With prepared, hash-bound inputs and a new output filename:
python -m src.parallel_source_v4.retrieval --data-root /authorized/data --fields runs/fields.json --queries protocol/queries.json --dense-cache runs/specter2-rankings.json --model-directory /authorized/models/reranker --model-manifest protocol/reranker-manifest.json --output runs/retrieval-results.jsonAll relative inputs/output are resolved under the data root; the model directory is explicit. Model-manifest model_revision must be a 40-hex revision and its file list must cover the entire installed snapshot. Verify actual CLI capabilities before adding protocol flags; the current candidate depth defaults to 50.
Acceptance and deliverables
- Exactly 10,000 frozen document IDs and 60 original queries are accounted for; missing source/model inputs block the full claim.
- No target insertion or query-dependent field repair. A metamorphic check that changes only target labels leaves candidate/rerank outputs unchanged. The current dense-cache query hash includes target_id: separate the ranking-input identity from evaluation-label identity, or explicitly rebind the same unchanged rankings for this test. Preserve both identities and do not mistake a stale-cache rejection for a ranking-independence test.
- Dense results bind to current fields, queries and model; stale caches fail. All channels and their absence are reported truthfully.
- Results include candidate recall at declared depths, union membership, post-rerank metrics, full miss classification and explicit f115/f076 outcomes.
- Compare like-for-like baseline conditions; distinguish implemented corrections, new runtime/model effects and genuinely measured quality changes.
- Publish scripts/protocol, sanitized metrics and immutable hashes; retain private corpus/query text and full judgments externally. Deliver a report stating residual limitations and follow-up issues.
Inputs and authority
Requires authorized access to the original corpus/query artifacts, source assessments, licensed local SPECTER2/reranker snapshots, compute budget and reviewers. Local runs need no paid provider calls. Respect model licenses and established data-transfer/spend limits; identify unavailable model access or compute budget explicitly. Unavailable models are NOT RUN, never substituted by a fake scorer. No graph writes are part of this experiment.
Tracking and completion dependencies
Parent: #12. Milestone: Paper extraction and GraphRAG qualification.
Completion prerequisites: #21. These are native blocked-by links; design, fixtures and other independent implementation work may proceed in parallel.
Required by: #23.
Activity
CompleteDotTech commentedon Sep 28, 2026
Implementation PR #28 merged with six hosted checks and verified merge commit
36a7f4f5e62be8dfdaba0739ca93b336664a8937, but this issue remains OPEN. The required 10,000-original-paper / 60-frozen-query retrieval trial, channel recall, actual reranking and source review have not completed. The two-document pinned SPECTER2 smoke is an execution check only (cache SHA-256de13b1b057e03120d8f85e89cbb32fb1c9d6a414b7b28ec84bc77a4bc3b66996). Full-corpus acceptance will be recorded separately after #21 fields are bound to the final extractor.CompleteDotTech commentedon Sep 28, 2026
Partial #22 implementation merged in PR #34 at
862a45757ce1d3458a6201f22e71f63391059958(signed head verified). It adds a frozen, hash-bound 10,000-paper/60-query trial wrapper, local SPECTER2/cross-encoder execution, full ranking readback, target-label permutation, supported without-dense ablation, and resource/integrity receipts.Verification: 49 focused tests passed; full local suite 723 tests, 718 passed/5 skipped; release validation PASS_WITH_EXTERNAL_GATES; package validator PASS; all six PR checks and three postmerge main checks passed. No 10,000-paper/60-query numerical run or relevance/source judgment has occurred. The issue remains open for a final-tree field rebuild, the real trial, f115/f076 page review, miss analysis, and remaining comparable ablations.
CompleteDotTech commentedon Sep 28, 2026
PR #41 merged as GitHub-verified
af89d02863da0ac8af19e2259a9a06c08b72d988. It closes the source-integrity gap where field verification hashed a PDF/image path and later reopened it for decoding: the builder now hashes and decodes the same source/page/image/native buffers. The adversarial path-swap test passes; local full validation ran 860 tests (855 passed, 5 skipped), package validation passed, all six PR checks and all three postmerge main checks passed (run 36474921906).#22 remains OPEN. The final 10,000-paper fields must be rebuilt on the final extraction method tree, then the frozen 60-query dense/rerank trial, source judgments, ablations and acceptance metrics must run. No retrieval-quality result is claimed by PR #41.
CompleteDotTech commentedon Sep 28, 2026
Corpus-policy and ablation hardening merged in PR #46, verified merge
c2b58d96dd3255e9baa1dd737155a5908b113d33. Six hosted PR checks and all three post-merge main jobs passed. Local official validation: 975 run / 968 passed / 7 skipped; 31 corrected focused tests and separate peer checks passed, including real-PDF/subprocess rejection of fabricated field eligibility and invented OCR evidence. The offline package check passed.The trial now freezes a reproducible corpus policy, rebuilds source-bound fields at freeze/start/end, and can measure page-channel and historical-field ablations with fresh embeddings. This is implementation and adversarial-fixture evidence only. No 10,000-paper/60-query model run or relevance judgment has been performed, and no retrieval improvement is claimed. A complete single-method converter assessment map for 10,000 papers is still absent (the available development converter set covers 200). Eligible OCR rows also need genuine producer assets. Native-only baseline remains a distinct available condition. Issue stays open until the real trial, source judgments, full denominator, and missing-input policies pass on the final code tree.
CompleteDotTech commentedon Sep 28, 2026
Measured a source-bound native-only lexical baseline on the full 10,000-document field file and the original 60-query set. Output: private
validation_v3_private/issue22-nativeonly-10k60-20260928-01/results.json, SHA-256edf780e890fa305b68a724adfdfdbfa676bae971c3502b2969cd15b306d7e09b. Field semantic SHA-256454ecbe24e18ccb7b3524d2a5a7a520df670eb2da34430624b3574559ed5db63; 10,000 fields, 9,986 searchable. Query file SHA-256dc521d6ad32440db2df130abc54f1771ff5aa845442ae2d62a1616e67b0ca8ac.At depth 50: field-weighted BM25 57/60, full-page BM25 59/60, union 59/60. f115 (target d04143) appears at field rank 14 and page rank 23; f076 (d19419) remains outside the union. Candidate-order top-1 52/60 and top-10 55/60 are not reranked. SPECTER2 status NOT_RUN, rerank 0 seconds, no model/provider/graph calls. The field build has 10,000 missing abstract assessments, so this baseline does not meet the repaired-abstract, dense, rerank, ablation, or judged-miss acceptance gates; those remain open.
CompleteDotTech commentedon Sep 28, 2026
Metamorphic label-independence check on the same 10,000-field/60-query lexical baseline: rotated all 60 target IDs while leaving every query text unchanged. The ranking-input SHA, ranking-output SHA, and complete per-query candidate pools are identical; evaluation-label SHA and scored details change. Private permuted result SHA-256: 7b76250ba70cf026ab018f12d5d79a1b102e632c731ccd323a9fd988daedb89d. This verifies the native-only lexical stage only; dense and rerank stages remain unrun.
CompleteDotTech commentedon Sep 28, 2026
Native-only 10,000-paper local rerank measurement (partial baseline, not the repaired-field qualification gate):
454ecbe24e18ccb7b3524d2a5a7a520df670eb2da34430624b3574559ed5db63.b96c10797edc8a1d2ad4dd22a450f85eee8acfea0e973741ba274c1be0e4a9c5; recorded ranking-input hashdc1012fe289d939c9172915f1c8047b0a53fa674360e30957fa5221252d1f6b7, ranking-output hashba94a932ed429b81c61077e42613d35b81a1b303f24ed925de496174173132f3, model-manifest semantic SHA-2567f3daae852a02f1c046068264d414b250fdaca28a7838fdf5afff4229a8b3288. Runtime 203.6 s including 178.4 s rerank, with zero paid API calls and zero graph writes. Corpus, judgments and full model files remain private.35fb98da91d4f049f717c765f6ec54699f7a1cc0a3b2e27471d041f4901e2d82. Fields, ranking inputs, ranking outputs, candidate pools and dense cache were byte/semantic identical; evaluation-label hash changed from51a2484e17bb1bb9c0e698340cd46cb796acb769197d6b3e3f22a7acebe5d167to4eb1cfb6d5cae227412a047cdd4d21ad1818b022c40a1b9c76145ae717e53265. Its known-item scores become zero because the targets changed, as expected. This checks that ranking is independent of target labels.Still open: locate or produce the authorized 10,000-paper abstract-assessment map, rebuild fields with it on the final tree, rerun embeddings and all stages under one frozen protocol, and review the f076 cover OCR/backfill against source without target-derived fields. These native-only numbers must not be represented as the completed repaired-field trial.
CompleteDotTech commentedon Sep 29, 2026
PR #60 merged as signed and GitHub-verified main commit
84c3c67e121bb9ec9e0041cb06565cc1fd7feaad; all six PR checks and all three post-merge main checks passed (run 36523231724). The redacted interim receipt isreview/retrieval_native_10k_interim_v1/receipt.json, SHA-2563c3c05510cd2c8191da1326fc3d39309550d7c0d0b4a8e09728132eb56eb39ab.The frozen 10,000-document/60-query native-field trial completed with terminal integrity and bounded peer readbacks passing. Current native title/body fields plus two unreviewed OCR cover bodies: 59/60 top-1, 60/60 top-10, MRR 0.991667. Archived native-only fields: 58/60 top-1, 59/60 top-10, MRR 0.975. f076 moved from absent to rank 1. Five target-label rotation pairs passed; 12 source errors remained in the denominator. The private terminal receipt SHA-256 is
194b44cbf40ffa7a317ca313791aab40a5a5ef598548e4a9915d5c3bf0925744.All 10,000 abstract assessments were missing, and the frozen title-role defect (#52) remains. The full V4 map (#53), final like-for-like comparator, and relevance judgments are outstanding. This is a partial baseline, not #22 qualification; leave the issue open. No paid calls or production graph writes occurred.
CompleteDotTech commentedon Sep 29, 2026
The repaired-title successor trial has completed and is published in merged PR #66, with the redacted source-bound receipt. #52 is closed.
The sealed run included 10,000 documents, 60 frozen known-item queries, fresh SPECTER2 rankings, and local cross-encoder scoring. Final source/code/runtime and driver readbacks passed; all five target-label permutations passed. Page BM25 contained 60/60 targets at depth 50, field BM25 57/60, SPECTER2 56/60, and the union 60/60. The full-channel rerank yielded 59/60 top-1, 60/60 top-10, MRR 0.9917. Removing the page channel yielded 56/60 top-1, 57/60 top-10, MRR 0.9417. Removing SPECTER2 left the aggregate full-channel scores unchanged. Repaired and archived titles had equal aggregate scores, although 14 title records and some rankings changed.
Status: partial native-field baseline, not #22 acceptance. All 10,000 V4 abstract assessments are absent (#53); 12 native extraction errors remain in the denominator; two local OCR cover bodies are unreviewed. Competing-result relevance judgments and a later independent query set are not complete. The receipt keeps known-item scores separate from broader relevance quality, and no graph writes occurred. Next gate: source-bound V4 assessment map for the full corpus, followed by a new frozen field/dense/rerank comparison and attributed relevance review.
CompleteDotTech commentedon Sep 30, 2026
Current delivered-code retrieval prerequisites
The full successor trial remains pending the genuine complete 10,000-document assessment map. The single owned recovery run is live and importing its 1,207 preserved rows; these imported rows are not new converter calls or a completed corpus measurement.
Completed readiness work:
23d014e2883ca85ab87e459e680ecace6e2a1c4001eb16e5d43fe12da8407585. Publisher cards at the exact pinned revisions declare Apache 2.0; separate metadata receipte7bdd112ec70170a3c0c762441424fa0b1cbc645045ab81f05abe359ef292fc2.bf4b97b38c573994b1a410d7314a9a8c19f0defdd89008b3c380ef775450f49a.b180d76b56f267f9dc8e22114540299414c6c3c9229919c0379842ef5425aa78; replay receipt355b29a4e486f312dab7581d48fd1514706020f68c62b502a05b4d4708a5d078.abstract_modetonative_assessmentsandassessment_methodtoparallel_structure_v4; OCR identity/mode are preserved. Policy hash46cbe0a64c5fbf94f2ff53ff2674b222616ae72fa238cff9834b1ff4e25d6b5f. The old abstract-disabled policy remains immutable.No full field build, dense embedding generation, reranker run or successor retrieval trial occurred in these readiness checks. The prior native-only ranking result remains a partial baseline with its title confound and missing relevance judgments preserved. Current competing-result review and final f115/f076 outcomes remain acceptance work. This issue remains open.
CompleteDotTech commentedon Sep 30, 2026
Pre-result relevance review readiness (qualification remains OPEN)
The prospective relevance rubric is now frozen before new retrieval result inspection, with source-first judgments, blinded reviewer packets, query-input/source-hash ordering independent of target labels, explicit misses/near misses and f076/f115 monitoring, competing-result comparisons, and unresolved/unassessable denominators. Review is attributed assistant review under the accepted protocol; it does not claim independent human validation.
Immutable identifiers:
2fba61be3beaef7b47771416d99039be2c770bf6905b901afd57fbcbd39004d6.2e70c1b3d5ec8475767ea25aa0e76902e74318a7bbaad5470a9e76dbe55464b8.e4793639bfca8c48c4d68cf76370d64d1dde691fcb61d5bf4cb4858e7016f2ac.4b83bd97b1fa2e9e368ab437d40e16e65b8a4751720b4ad8b6317f9d916a11b5.Seven synthetic packet-mechanics tests passed in peer and owner runs. Reviews corrected missing all-60 private target-rank accounting, arm status/provenance, ordering-collision rejection, output confinement, packet byte-hash binding, and verification of every duplicate source ID's actual PDF/page/render bytes. A portable public implementation is being prepared in a separate checkout. These are tooling checks; no real reviewer packets or relevance judgments have been produced.
The live assessment continuation has imported all 1,207 sealed prefix rows and is re-verifying the prefix before new conversion. Imported rows explicitly declare no new model conversion. The full 10,000-document assessment map is still absent, so the reviewed full-corpus field-build/trial commands remain disabled. The original run cutoff and ownership remain intact.
Remaining acceptance: complete the authentic full-corpus assessments, rebuild and verify all 10,000 fields, execute the pinned models on all 60 original queries with stage metrics/ablations and a real label-independence check, then produce blinded packets and seal attributed source judgments. Prior partial retrieval scores remain preserved and do not establish this acceptance.
CompleteDotTech commentedon Sep 30, 2026
Progress checkpoint: the existing full-corpus producer is live under its original cutoff. The latest observed state accounts for 1,411 rows: 1,207 preserved sealed imports and 204 genuine new conversions. Two bounded worker rotations were observed; earlier workers exited and a replacement was live. Conversion success does not automatically make an abstract eligible for retrieval.
Portable relevance-packet tooling now has scoped attributed assistant review PASS and 13 passing synthetic tests, including Windows junction rejection and a changed-result-during-copy hold. Legacy package validation, isolated installation and CLI checks pass. Full repository validation is still running; publication and the source archive audit remain pending.
The publication checkout retains byte-identical source for all 83 extraction and 124 image-path files bound by the accepted 200-paper receipt. This is an identity mapping, not a new empirical run.
The complete 10,000-document map, real 60-query trial, relevance judgments and semantic graph/answer qualification remain outstanding. This issue and the epic stay open. No independent human review or empirical accuracy is claimed by the tooling checks.
CompleteDotTech commentedon Sep 30, 2026
Delivered: PR #89 merged as
4400fb9cbc94109038f08a7f1382138f01f77ced. Both hosted CI runs passed all Python 3.11/3.12/3.13 jobs on reviewed signed head43a8b0f51ba1e2592e726dd763ebd5fd2828c5f8; the merge retains exactly the reviewed tree. GitHub verifies both signatures. All 13 packet tests passed after merging. Default-branch CI is running.Final attributed assistant release review covers all 24 changed files and the generated evidence. Full validation: 1,328 total, 1,320 PASS, 8 optional skips, zero errors/failures. The 585-member source archive passed audit. Synthetic demo timestamps/IDs and benchmark latencies changed; two adaptive retrieval fingerprints changed without scientific metric or policy outcome changes. These are synthetic checks, not real GraphRAG qualification.
The full-corpus COMPLETE-map binder additionally passed an isolated authored 10,000-row positive case and five corrupt-input cases, including mutations with matching updated hashes. No real corpus authority or field build was created by that check. The next live handoff still requires the producer's genuine sealed COMPLETE map and ledger.
The corpus producer remains live: latest observed 1,527 rows (1,207 sealed imports and 320 new conversions). Its original cutoff remains in force; an extension request is pending. Full retrieval, source relevance and semantic graph/answer gates remain outstanding, so #22 and #12 stay open.