Wire source-bound abstract assessments into retrieval field generation #21

Problem and evidence

build_fields verifies source/page/image/native files and calls extract_fields with optional title review and OCR cache. It never passes abstract_assessment. extract_fields already supports a sealed assessment with a proposal and source spans, but the field-building entry point leaves every abstract field empty.

This is an integration omission, not a request to replace working title filtering or candidate channels. V4 already excludes boilerplate such as “Graphical Abstract”; the September 28 read-only probe selected f115's real title. The full-page BM25 candidate channel also already exists. f076 still has no native searchable text and requires source-bound OCR or image review. No new 10,000-paper ranking result was established by that probe.

Scope and implementation

1. Add an explicit input mapping from sample/document IDs to immutable abstract assessments in the field-builder API and CLI. Specify its schema and method/version selection; reject duplicate/unknown IDs, ambiguous method selection, malformed receipts and mismatched document/page identities.
2. Pass the selected assessment to extract_fields only after integrity and provenance checks. Verify the proposal's spans against the exact native representation selected for this run; a self-consistent assessment hash alone must not establish source location. Bind the native representation and extractor/environment identity in the output receipt.
3. Resolve the current evidence mismatch risk: the builder verifies the cached native file's hash but re-extracts native lines from page.pdf. Either consume and validate the pinned saved representation or explicitly produce/version a new representation and revalidate dependent assessments. Never silently combine cached assessments and a different native extraction.
4. Preserve per-field provenance, candidate/review state, source/page/image identity and assessment hash. Retrieval indexing must retain retrieval_only=true, eligible_for_jev=false, and unreviewed status for selector proposals.
5. Keep held/error/absent/partial candidates distinguishable from available abstract fields. Document which reviewed OCR/title fields may aid discovery and why discovery does not qualify graph evidence. Consume any new image-evidence contract explicitly when available; do not invent native spans.
6. Hash the complete resulting fields, invalidate downstream dense caches after any field change, and write a new immutable output. Inputs must remain query-independent; neither query text nor known targets may influence field selection.

Reproduction and tests

Current CLI, before this change; paths below are placeholders under an authorized data root:

python -m src.parallel_source_v4.fields --data-root /authorized/data --manifest corpus/source-views/manifest.json --output runs/fields-before.json
python -m unittest discover -s tests -p 'test_parallel_source_retrieval_v4.py' -q
python -m unittest discover -s tests -p 'test_parallel_source_io_v4.py' -q

There is currently no CLI argument for assessments. Add and document it rather than claiming the example above exercises assessment wiring. A regression test should invoke the actual CLI with a manifest and a valid sealed/source-bound assessment; directly calling extract_fields alone would miss the original defect.

Acceptance and deliverables

• An eligible matching assessment produces the exact expected abstract through the actual field-building CLI, with assessment/native/source identities in its provenance.
• Wrong source/page, modified spans or text, stale native evidence, duplicate mappings and unsealed assessments fail explicitly; held inputs cannot silently become complete abstracts.
• Reusing a dense cache from pre-change fields is rejected by the existing stale-field check; unchanged immutable input reproduces identical content hashes.
• Public synthetic fixtures exercise boilerplate-title exclusion, title below the old search region, no-native-text OCR fallback and candidate-only status. Real f115/f076 readback is reported without publishing their private text.
• Field generation for the authorized full corpus produces explicit missing/held/unsearchable counts and immutable provenance. Do not claim retrieval quality from this step; link the separate retrieval experiment.
• Deliver code, schemas/input examples, CLI documentation, targeted test results and a sanitized field manifest with counts/hashes.

Inputs and boundaries

Obtain the existing authorized corpus manifest, pinned native evidence and corresponding assessments from the data owner; the public repository intentionally excludes private PDFs and full assessments. Core wiring can be implemented and tested with synthetic fixtures independently. No paid API is required for this change. Field indexing does not authorize evidence admission, Jev calls or graph writes. Preserve old fields and experiment receipts.

Tracking and completion dependencies

Parent: #12. Milestone: Paper extraction and GraphRAG qualification.

Completion prerequisites: #13. These are native blocked-by links; design, fixtures and other independent implementation work may proceed in parallel.

Required by: #22.