Add versioned notation and source-boundary fidelity metrics without rewriting historical scores #14

Problem and observed evidence

The current scorer strips nonalphanumeric characters before comparison. It assigns perfect precision, recall and boundary agreement to these materially different expressions:

• x = 1 versus x != 1
• x + y versus x - y
• 10^5 versus 105

This proves a metric blind spot; it does not establish a notation error in every accepted paper. The frozen v2 baseline has 91 historical text98 successes but 90 historical boundary-plus-text98 successes: f195 appends an extra final footnote marker. The prose in the public report must distinguish these definitions.

Code entry points

• canonical and compare.
• Per-case scoring and summary.
• Frozen historical scorer; preserve its behavior for legacy replay.
• Known-set report currently describes the table as requiring a valid boundary although its baseline count is the legacy 91.
• Source review/freeze validation.

Required implementation

1. Preserve the legacy text_match_98 and boundary_and_98_match definitions and historical receipts under explicit metric versions. Add separate metrics instead of silently changing their meaning.
2. Define notation-aware comparison preserving meaningful operators, negation, superscript/subscript structure, signs, units and equation order. Specify permitted normalization (for example reviewed Unicode/font-equivalent forms) with positive and negative fixtures. Do not claim semantic equation equivalence from punctuation stripping.
3. Measure boundaries against source-reviewed start/end spans or image regions and section ownership; surface extra prefix/suffix material, footnotes, metadata and body contamination independently from approximate text agreement.
4. Represent notation and boundary review as pass/fail/unresolved/not-applicable with attributed evidence. Separate source location, transcription fidelity and review status. Unresolved scientific notation must not count as verified correct.
5. Extend summaries with metric-version identifiers, eligible denominators, false proposals, complete-abstract recall, reading-order and notation errors, review workload, and skipped/unresolved cases. A matching withheld candidate remains withheld.
6. Correct the public report's 91-versus-90 explanation without changing the historical dataset or lowering the 98% threshold. Correct f195's output in the selector/boundary work and measure the improvement here.

Reproduction

A minimal synthetic demonstration, with no private data required:

python -c "from trace_gc.pdf_source_parallel_v4 import compare; print(compare('x = 1', 'x != 1')); print(compare('10^5', '105'))"
python -m unittest discover -s tests -p 'test_parallel_source*v4.py' -q

After implementation, run versioned metric tests and the integrated 200-paper gate; retain both legacy and new results for every case.

Acceptance criteria

• The three counterexamples above fail notation fidelity even when historical text98 remains true.
• Meaning-preserving rendering/whitespace differences allowed by the documented policy pass; sign, exponent, equation-order and unit changes do not silently pass.
• Authored contamination tests distinguish exact abstract boundaries from approximate text success.
• Frozen v2 remains 91 legacy text98 / 90 legacy boundary-plus-text98; f195 is explicitly traceable without publishing its paper text.
• Unresolved notation and missing review evidence remain visible and cannot be reported as certified fidelity.
• Full 200-case summaries retain absent/partial/error/withheld denominators and metric identities; an all-abstain selector cannot appear to have perfect measured coverage.
• Historical receipts remain byte-preserved; updated documentation names the evaluated property rather than implying scientific fidelity.

Research informing the design

olmOCR-Bench's tests separate text, ordering and mathematics; OmniDocBench evaluates document components separately. These are implementation patterns, not evidence that TRACE-GC already passes.

Private evidence: an authorized maintainer can resolve validation_v3_private/diagnosis-20260928/receipt-audit.json and ROOT_CAUSE_AND_SOLUTIONS.md beneath the external test-data root. Latest results.json SHA-256: 661964b8db10bf2bd806a95a0119f4492dcabf72e759fff521993f702f016b0d; historical: f9ee4f438c8cae7ce192970f7ff172ecbe7c24b3a383d48fe62cf75da7f4102b. Obtain authorized local access to PDFs, frozen labels, converter caches and the f001–f200 relative source map following the reproduction guide. Keep those assets and full assessments outside the public repository; publish only redacted aggregate results and hashes. Missing access is a blocked quality run, not a passing substitute.

Scope

Metric implementation can proceed independently of selector repairs. Use versioned/additive evaluation so alignment behavior is not accidentally changed by replacing a shared canonical() helper.

Tracking and completion dependencies

Parent: #12. Milestone: Paper extraction and GraphRAG qualification.

No mandatory issue prerequisites; implementation can begin using the documented inputs.

Required by: #18, #19.