Repository navigation
Implement and validate a real paper-to-graph-to-cited-answer pilot #23
Description
Outcome and current gap
Deliver a reproducible application harness that takes authorized real papers through reviewed evidence admission, graph synthesis and durable publication, graph retrieval, and generated answers with verified supporting citations. This is the missing integration test for the paper-to-GraphRAG objective.
At the inspected revision, pdf_admission.py prepares a reviewed first-page Jev payload but does not register it into the graph runtime. GraphRAGQueryService.query returns CONTEXT_ONLY, not a generated answer. The CLI exposes synthetic demos and validation, with no real-paper pipeline command. Existing GraphRAG documentation describes injectable components and application boundaries; the first-page 200-paper benchmark cannot establish this full workflow.
This pilot is related to domain-graph epic #4 and production deployment issue #11. It does not duplicate their classification/storage work, and selecting a production backend is not a prerequisite for a clearly labeled local pilot on the durable reference backend.
Implementation and bounded protocol
- Freeze an authorized real-paper cohort and internally source-reviewed questions/expected evidence before running. Include a native-text paper, structured or multi-paragraph abstract, an image/corrupt-text case when that route is ready, and evidence beyond page one. State the document count, work identities, rights, budgets, coverage and pass criteria. Include unsupported questions and conflicting evidence where the corpus supports them.
- Implement a separate document-scope ingestion contract with source/page/image identity and reviewed spans. Keep physical-first-page benchmark semantics unchanged; a later-page abstract must not turn a first-page “absent” result into a “complete” benchmark success. Map reviewed native/image-derived text into immutable catalog source/evidence records with exact offsets in the declared representation and traceable raw-document lineage.
- Add the missing application orchestration: source admission, candidate generation, actual semantic observations, compilation/resolution, review, and operation-authorized publication through
CompilerService. Persist run identity, plans, receipts, graph journal and budgets. No direct SQL graph injection or synthetic fixture responses may satisfy the real semantic path. - Use application-owned trust and reviewers, the selected exact policy and one durable graph authority. A disposable local
SQLiteReferenceBackendpilot is acceptable with explicit scope; it is not production backend qualification. Close/reopen the database and verify state and provenance through fresh connections. - Retrieve context using authenticated application-owned access scopes. Add an answer-generation adapter using an approved pinned provider or authorized local model; record prompts, model/tokenizer identity, budgets and responses. Associate each material claim with exact selected evidence citations. Check citation existence, source identity and actual claim support; mechanical citation matching alone cannot certify entailment.
- Add resumed execution, repeated immutable requests, source revision/withdrawal and permission-change scenarios. Stale plans/reviews must fail and require refreshed approvals. Default answers must exclude withdrawn/denied evidence; historical inspection must preserve current permissions. Unsupported questions must abstain with a clear reason.
Acceptance and deliverables
- One documented supported command or service workflow runs the actual authorized cohort end to end, pausing for required review and resuming from durable state. Implement/document that interface; none currently exists.
- Each accepted answer claim traces through citation → source evidence → PDF page/crop/representation, and graph assertions trace through transaction/plan/resolution/observations. Report incomplete chains explicitly.
- Attributed internal review measures extraction completeness, evidence quality, graph correctness, retrieval coverage, answer support/citation correctness and abstention separately, using predeclared criteria. Assistant labels and reviews remain attributed; no independent-human claim is made.
- Fresh database reopen reproduces graph state and historical receipts; retries do not duplicate publication. Source changes, withdrawal, rejected review, missing provider output and interrupted execution are exercised.
- Public synthetic fixtures cover mechanics, and a separate hash-bound real-run report proves the real-paper route. Synthetic demo output or cached fabricated semantics cannot close this issue.
- Deliver harness, contracts/configuration examples, operator/reviewer runbook, sanitized manifests and metrics, cost receipt, failure/recovery evidence and linked residual issues. Run required repository release/package gates for shipped changes.
Inputs and dependency boundary
Requires an authorized corpus, attributed internal reviewers, application-enrolled trust/source authority, an isolated persistent target, and approved model/provider/tokenizer access plus spend and data-transfer limits. Complete local implementation and synthetic mechanics while those inputs are arranged; identify any genuinely missing input precisely. The image route is needed for the declared image case, but native-paper pilot work can proceed independently. Harness implementation and native-paper trials may proceed in parallel. Final qualified delivery under this epic depends on unseen internally reviewed extraction qualification and the retrieval trial; bind the selected configuration to their receipts.
Deployment requirements remain applicable to production claims and automatic consequential publication. No permission to use production credentials, send private papers externally, bypass review, or publish into a production graph is implied by this issue.
Tracking and completion dependencies
Parent: #12. Milestone: Paper extraction and GraphRAG qualification.
Completion prerequisites: #20, #22. These are native blocked-by links; design, fixtures and other independent implementation work may proceed in parallel.
Activity
CompleteDotTech commentedon Sep 28, 2026
Partial #23 implementation merged in PR #33 at
cc809f73204c57290d424f68c36013260c428c5f(GitHub verified signature). This adds reviewed document-page source lineage and exact span readback, held cited-answer draft/review contracts with current graph/source/access checks, and an optional hash-pinned offline local Qwen adapter. The synthetic cited-answer smoke producedREVIEW_REQUIREDonly; it is not real-paper qualification. Local merged-tree validation: 704 tests, 697 pass, 7 skip, 0 failures/errors, 0 provider calls/production graph writes. All six PR checks passed; postmerge main CI is pending.The issue remains open. The durable orchestration command, authentic semantic path, authorized real-paper cohort, graph publication/reopen, retrieval, claim-support review, withdrawal/recovery checks, and dependency receipts for #20/#22 are still required. The latest integrated #19 200-paper run failed preservation (20 of 91 prior correct cases lost), so no end-to-end acceptance is claimed.
CompleteDotTech commentedon Sep 28, 2026
PR #37 merged the signed durable pilot service on 2026-09-28 (merge
ed2b5638f8b46a4149173f4e8d173a79b976c5eb, GitHub verified). Six hosted PR checks passed. The integrated local run passed 792 tests (785 pass, 7 skip; no failures/errors, provider calls, or production graph writes); package validation passed. Post-merge main CI is running at https://github.com/Jev-Engineering/jraphyte/actions/runs/36466144652. The service supports durable, review-gated page-source → semantic response → graph publication → cited answer steps with explicit pause/recovery and current ACL checks. It has not executed a genuine semantic model or a real paper-to-answer pilot. A compatible local semantic adapter, authentic model receipt, final #19/#20 paper qualification, #22 retrieval trial, and source-first internally attributed answer review remain required. This issue stays open.CompleteDotTech commentedon Sep 28, 2026
Post-merge main CI run 36466144652 has now completed successfully on Python 3.11, 3.12, and 3.13. The remaining real-pilot gates in the previous comment are unchanged; #23 remains open.
CompleteDotTech commentedon Sep 28, 2026
PR #39 merged the signed, distinct local Qwen semantic profile at verified merge
38576cb6e7f1b6b9f169ad35ec303c6109fb9704. All six PR checks passed; integrated local validation passed 837 tests (830 pass, 7 skip), package validation passed, and post-merge main CI is running at https://github.com/Jev-Engineering/jraphyte/actions/runs/36470754986. The profile pins actualQwen/Qwen3-4B-Instruct-2507snapshot/revision, prompt, tokenizer, runtime and implementation identity; CHOICE output is strictly parsed as uncalibrated model judgment, with original bytes and explicit errors retained. Legacy Jev wire behavior remains unchanged. An already exposed f097 paper/claim has two attributed source-first reviews sealed before any model output. A strict, isolated model snapshot and runtime are prepared, but the model has not been loaded or called because available free RAM is below the coordinated 20 GiB headroom. No real semantic observation, graph publication or cited-answer result is claimed. #23 remains open.15 remaining items
CompleteDotTech commentedon Sep 30, 2026
Seven-claim pilot: actual compile and same-approval replay passed
The new source-verified pilot exported one real semantic request pack containing all seven frozen candidate claims, backed by the eight freshly reviewed admitted source surfaces. The same externally pinned compile-only release replayed successfully and preserved the exact result bytes.
45ce58a63c55f46051a2fa51d025d3106a10237069a3de4bb437a10504d94f71; independent static/API review passed and root reran 11 focused tests.3c877e458fdb73ac58fc4305a05d3d81e74060414b9ff17adaae6895580ca987.ce27a0919581e36fd0601028f36e615e003263afeb64110d38f3c29742efc04e; pack:6aea39d273ac4c0d0104bbf22f2ec3897e28e8cb02812cbe7026a86dc441274e.WAIT_SEMANTIC_RESPONSE, with external execution required and automatic retry disabled.The reviewed controller preserves the compound q08 question and its two distinct source facts, and its access-grant recovery handles both a crash after atomic replacement and a completed same-release replay. The focused fixtures used authored synthetic data and qualify API mechanics only.
The current 200-paper resource guard observed 6.642, 6.794 and 6.426 GiB available across three samples. It failed the unchanged 10 GiB floor; the 200-paper recovery remains unlaunched. Actual model execution also remains gated by its unchanged 20 GiB sustained floor and shared-worker priority. Source-bound operation-map review is the next independent task. This issue remains open pending real semantic observations, reviewed publication, retrieval, generated cited answers and the remaining acceptance scenarios/dependencies.
CompleteDotTech commentedon Sep 30, 2026
Acceptance audit: frozen semantic intent and remaining controller repairs
The actual seven-claim run has now frozen semantic intent
357b4bfe178310b70ad92535f9c4d5f99b7ff94330bb905e9de2f52c424bf52a, bound to the actual compiled pack and independently reviewed operation map59696bcbb45e5fca0dd7d7c17bc3efdbd1a431790f23126aaeccd46c3c85c8aa. No inference attempt has started. The graph remains at version 1 with eight sources and zero assertions; the one model-call budget reservation is not an executed model call.The full acceptance audit identified two gaps in the private application controller, now assigned for versioned repair:
Completed semantic reconciliation after publication. The v7 controller calls
PaperPilot.semantic_execution_contextbefore checking whether the semantic response was already durably imported. That API correctly requires the frozen compile snapshot to remain current. Publication changes graph version 1 to 2, so reconciliation raisesPILOT_STALEfor an otherwise completed import. An independent disposable seven-assertion API fixture reproduced this causal sequence. Fix: verify completed imports against their exact historical checkpoint/request/raw response/execution attestation/import result and publication journal, without reimport or model retry. Keep fresh-context checks for pending imports; missing or inconsistent historical lineage must hold. Preserve current access checks. Independent finding receipt:42a84130ce42a58f11e8a7f0b8706cfed18390f203a3e1246ba7370ccac2b558.Postpublication source-change scenarios. The earlier revision/withdrawal/permission tests had zero assertions. They do not establish invalidation of a published seven-assertion graph or generated answers. A separate versioned scenario controller must use isolated snapshots of the eventual qualified parent graph and preserve its journal, checkpoint, source reviews, answers and budget history. Withdrawal and denial must invalidate affected assertions and stale answers/contexts. A genuinely changed PDF requires a separately authorized successor document frame and run; the frozen parent frame cannot be rewritten or relabeled. The successor must retain explicit parent lineage and cumulative budget limits. Same-PDF transcription correction is not a substitute for changed-document revision.
The aggregate evaluator is also being completed to check all eight frozen questions, seven genuine assertions, exact citation/source chains, six supported-query top-five results, two reasoned abstentions, the edition discrepancy, and every required recovery scenario. Missing model outputs and incomplete chains remain
NOT_RUN/HOLD/FAIL, with honest denominators. Authored synthetic fixtures prove mechanics only.No frozen reference, numeric target, resource floor, parent run or legacy score is changed by this work. Full real qualification and dependencies #20/#22 remain outstanding; #23 remains open.
CompleteDotTech commentedon Sep 30, 2026
Historical reconciliation repaired; aggregate evaluator remains held
The versioned reconciliation repair now passes scoped independent static/API review. Completed imports are checked against their original compile request, raw response, signed execution/import records, preexisting observations, reviewed publication chain and exact graph journal. Reconstruction uses a scratch catalog and cannot create observations in the live catalog. A graph-version-2 release requires the original publication artifacts.
Reconciliation receipts now use externally approved request IDs. Same-ID replay is immutable; changed evidence under that ID rejects. A separately approved new ID can record authentic recovery while retaining the earlier missing-output HOLD receipt unchanged. A disposable authored-synthetic PaperPilot fixture covered that transition and adversarial lineage failures. Controller SHA-256:
c17f510ff54d35ea527f803378aa00256538b78ad4e202ffee200e63bfd744e3; final fixture:6cfed56cd60c1af23d950608d14d4826d84298e012c163b67f7cf39b78f4025e; independent review:53166e21011fd37e3fbf7f4db669be644318bf70b527bbe497da340aacfe7462. This is mechanics evidence, not a real inference or release authorization.The independent aggregate-evaluator review found additional P1 evidence-binding defects: pinned answer bytes were not replayed against authenticated imports/checkpoints; abstention judgments were incomplete; and extraction/semantic chains relied on insufficient ledger checks. Review receipt:
1bc14f3b35317d128168363fbdaed666603517dde8810c499588fbc854ec3733. Versioned repairs and actual-shaped negative fixtures are underway. The evaluator remains DRAFT/HOLD and cannot qualify the pilot.The postpublication contract is also being corrected to preserve all eight actual answer executions/imports while requiring six supported signed reviews and two separately attributed abstention judgments. Permission/withdrawal forks retain original PDFs; the changed-document revision fork has its separately declared frame. Neither a real fork nor a scenario stage has run.
The actual parent remains graph version 1, eight admitted sources, zero assertions and zero executed model calls. Full current-source 200-paper, unseen, full retrieval, real semantic/answer and lifecycle acceptance remain outstanding. #23 and its prerequisite issues remain open.
CompleteDotTech commentedon Sep 30, 2026
Delivered lifecycle service and remaining empirical gates
PR #88 is merged at
d908283d94d44c22b8427b5ad463493424c5c967. The merge tree equals the independently reviewed, tested signed head. Local and GitHub signature verification passed.f495efd841d5684b7dd498f118eda355a0b74baf25e662e779f57433612aea91.ENOSPC; the passing run used a dedicated second local volume with unchanged source identity. This observation does not establish the cause of older converter failures.The genuine eight-source extraction replay remains separately recorded. Actual semantic inference, reviewed seven-assertion publication, eight-question answer assessment, and all nine real recovery criteria are still pending. Reviewed private compatibility, missing-output and archive-replay components prove authored API mechanics only. The current real parent remains held before graph/answer completion.
The unseen qualification (#20) and full retrieval trial (#22) also remain pending. A hash-verified input mirror is prepared for the unstarted current-code 200-paper rerun; its original inputs and prior results are preserved. That run retains its resource and scientific acceptance requirements. This issue and the qualification epic remain open.
CompleteDotTech commentedon Sep 30, 2026
Acceptance state correction and recovery work
Reopening this issue because its acceptance checklist remains unmet. The merged lifecycle service is implementation evidence; it does not qualify the real paper-to-answer pilot. The latest closure had no accompanying acceptance report or closing commit. Prerequisites #20 and #22 remain open.
No empirical denominator, model-call cap, scientific target, legacy result, or approval has been relaxed. The issue remains open until the real-run report and its completion dependencies satisfy the documented criteria.
CompleteDotTech commentedon Sep 30, 2026
Recovery integration update
The full authored successor test reproduced a causal retrieval failure: a fresh successor checkpoint has no parent
REVIEW_SOURCErows, while inherited graph citations retain the original source IDs.PaperPilot.retrievetherefore rejects the inherited sources. Copying checkpoint rows or inventing reviews would violate lineage.A versioned controller now uses current and historical GraphRAG service queries with detached catalogs/meters and signed exact results. The focused integration test passes on a genuine public fork, covering six supported answer chains, two abstentions, dual-source edition evidence, signerless readback, forged quote rejection, and unchanged parent state. The scenario suite also passes. This is authored mechanics evidence and service-level retrieval; it does not prove real cohort qualification or a PaperPilot retrieval checkpoint.
The interrupted withdrawal classification repair passed scoped peer review. Acceptance still needs authentic durable witnesses for rejected/stale review attempts and the actual interrupted-publication boundary. Final output JSON alone does not prove those attempts occurred.
The relocated evaluator remains under review: the new draft captures the bytes it hashes and consumes, with nine focused tests passing. Its complete artifact manifest and helper trust boundaries still need verification.
Real empirical acceptance remains outstanding. Keep this issue open; these checks do not substitute for the real paper/model run or the full nine recovery scenarios.
CompleteDotTech commentedon Oct 1, 2026
First source preparation stopped before opening the pilot
The owned first-source-prepare attempt exited with
IMAGE_RENDER_MISMATCHduring source preflight, before Pilot OPEN, source review, admission, or any model call. The original failure, fixed cutoff, and no-retry decision are preserved. Root and separate assistant readbacks found the staged metadata unchanged, the historical target files unchanged, and no new pilot state, private keys, review material, or prepare receipt. Full post-failure runtime closure and complete historical process ancestry were not established.Read-only diagnosis isolated the mismatch to three renderer version fields (PyMuPDF, MuPDF, and Pillow) on two image routes. The checked source PDFs, transcriptions, page and crop PNGs, and rendered pixels match their earlier evidence. The strict verifier correctly rejected the changed renderer provenance. The earlier candidates have not been edited.
Fresh candidates from the current renderer now have separate byte-bound lineage. An attributed assistant performed a source-first visual readback; this is not an independent-human review or an admission decision. Candidate review fields remain unset, and the notation anchor remains held. A disabled replacement protocol and run namespace use the approved external task/data roots, including the corrected D-root asset contract. Final mechanical review, fresh attributed image review, exact protocol and source approval, and a separate owned release are still pending. The failed attempt does not authorize a restart or extend its original cutoff.
The earlier historical provider outcome remains unknown and unreattempted, with its one reserved call counted. No new source admission, model call, graph assertion, generated answer, or recovery qualification is claimed. #20 unseen extraction qualification and #22 full retrieval acceptance remain unmet; #23 stays open.
Evidence: root failure readback
018b84e1de905febb2decf7badac7468b582d5451d13cb4f1a62a6e395e72239; independent corrected failure readbackdb94182c25fb38f50e4c7d3d761932fe30d20007b97faf689c9d7643f45d3e10; renderer diagnosis2c7fa62385c5c28db3782ba5878dd4449c907f5e5ceb2d08bf12f59504df0f9d.CompleteDotTech commentedon Oct 1, 2026
Source preparation remains stopped after a missing runtime dependency
The single owned V14 source-prepare attempt exited before Pilot OPEN, source review, or admission. Its frozen runtime omitted a required mapping helper: the source controller's gate adapter called that sibling module and stopped at the path check. The root and independent readbacks found the 35 frozen runtime files, four staged metadata files, and seven historical target files unchanged. No pilot state, private keys, prepare receipt, or source-stage lock was created. The earlier failed attempt remains preserved.
A versioned, disabled V9 bundle now includes the missing helper and pins a 36-file runtime closure. A separate captured read-only gate check exited successfully: it verified the historical source evidence and the reviewed current-code mapping, including 3,408 historical inputs, 87 unchanged measured methods, and the one known phase helper image difference. This is source-evidence code mapping, not a current literal acceptance receipt, a source-prepare PASS, or authority to admit a source, run a model, or alter the graph.
The original source-prepare cutoff has expired. Neither the failed V14 attempt nor the V9 read-only gate grants a retry or extends that window. Any later source action needs a separately reviewed release and window. The historical provider outcome remains unknown with its one reservation counted and no retry. #20 unseen extraction qualification and #22 full retrieval acceptance remain open; #23 has no generated-answer or recovery qualification claim.
Evidence: V14 root failure readback 712372bd315c7d2fc1ffe974a673ad3e1658ba0df41e4cd8e99842145c1b1025; disabled 36-file handoff 772bff545d72d687d3e3f6225f29b5ce497b8bd16a319513ba7bba39d338e2ee; read-only gate capture c81f161fa2f26021fd05a659908ed3941c68e25a9c79aeda9223eb3b23992844.
CompleteDotTech commentedon Oct 1, 2026
Runtime reconciliation: source preflight passes; pilot remains unqualified
The actual successor source attempt stopped at the omitted
fresh_gate_v3_mapping.pybefore source preparation. After repairing that dependency, read-only pre-OPEN checks identified three additional mismatched bindings:phase_transition.pyreproduces the original identity exactly.The prepared successor uses the existing pinned semantic environment without modifying its installed packages. Both image candidates were regenerated through the actual artifact verifier. Their original PDF, native text, transcription, page PNG and crop PNG hashes are unchanged. The versioned semantic profile changes only its implementation identity; model revision, tokenizer, prompts, generation settings and numeric gates are unchanged. The full pilot identity is rebound explicitly through its consumers.
Read-only checks now pass together in that one interpreter: three exposed PDFs, seven native anchors, two image anchors, profile/runtime equality, tokenizer assets and construction, required imports, clean delivered code and full pilot identity. The native f113 evidence remains withheld. These are pre-OPEN checks, not successful source preparation or end-to-end acceptance.
Evidence identities:
1e53f6aa85d2290412f7c39d3bd2bc7ab9c7a5311313c5586686c2f777f73d0b.132e91cfed7d1eed7f7001e60fac8623feed13b9287a8721acd4efd6cd49fa0d.c7207a7b3b53116156861203276960ef1c5642cce7ca0032eeb12f77dcff82e3.cbb38f883d6a28f1125957c6500cdda52fdbb55bccd88e8fd1409d856daa776c.86c9039cce7de8ef73e9215ec8f46235f3e2c2f03f84fe965c544dc04f9561cc.c48351ea6fb993b49e1aee140081397cd18bb528b69ae345df29d4508e513868.The environment byte inventory is captured; final execution-binding review is in progress. The predecessor source window remains expired and its failed attempts are preserved. No successor source stage, private run keys, model call, graph assertion or generated answer has been created. The last host check recorded approximately 2.3 GiB free RAM against the frozen 20 GiB model-loading prerequisite. Issues #20, #22 and #23 remain open.