MyOpenCRE mapping: reuse Module C Librarian retriever/reranker (RFC TODO 3)

[northdpole]

Context

RFC #876 TODO 3 defines application/utils/mapping_job.py with bi-encoder → pgvector top-20 → cross-encoder rerank. GSoC Module C (The Librarian) implements the same pipeline:

• C.1 candidate retriever — week_3: Module C (The Librarian) — C.1 candidate retriever (in-memory + pgvector) + pipeline switch #937 (approved)
• C.2 cross-encoder reranker — week_4: Module C (The Librarian) — C.2 cross-encoder reranker #957

Maintainer decision on #876: do not build a second parallel mapping stack. Wire MyOpenCRE run_mapping to reuse Module C components where possible.

Requirements

• Implement run_mapping by calling librarian/candidate_retriever + librarian/cross_encoder (or shared interfaces) instead of duplicating encode/rerank logic
• Map RFC user_standard_sections status enum (AUTO_MAPPED, NEEDS_REVIEW, etc.) to Librarian confidence thresholds — document mapping in RFC or inline
• rq job remains in mapping_job.py but delegates retrieval/rerank to Librarian modules
• Tests: fixture upload produces deterministic status counts; USER_CONFIRMED rows never auto-remapped (RFC constraint)
• Coordinate merge order: Module C W1–W3 (week_1: Module C (The Librarian) — RFC contracts + eval harness + golden dataset #922 → week_2: Module C (The Librarian) — C.0 input boundary: SectionValidator + ExplicitLinkResolver #925 → week_3: Module C (The Librarian) — C.1 candidate retriever (in-memory + pgvector) + pipeline switch #937) before or in parallel with MyOpenCRE TODO 3

Out of scope

• Full Librarian graph-write / HITL flow (Module C C.3–C.5)
• Per-user embedding storage (RFC explicitly forbids)

Related

• docs/rfc/user-auth-myopencre.md TODO 3
• Add support for users #586, Module C PRs week_1: Module C (The Librarian) — RFC contracts + eval harness + golden dataset #922, week_2: Module C (The Librarian) — C.0 input boundary: SectionValidator + ExplicitLinkResolver #925, week_3: Module C (The Librarian) — C.1 candidate retriever (in-memory + pgvector) + pipeline switch #937, week_4: Module C (The Librarian) — C.2 cross-encoder reranker #957

[Glitch-On]

@northdpole https://github.com/northdpole, https://github.com/northdpole I'd like to take this — the merge-order blocker is now clear #937 #937 #937 and #957 #957 #957 are both merged to main , so this is unblocked and a natural next step.

My read on the approach:

Implement run_mapping in application/utils/mapping_job.py by composing the existing Librarian primitives directly rather than re-implementing them: build_retriever(...) from candidate_retriever.py for the top-20 shortlist (RetrievalAudit.candidates[]), then CrossEncoderReranker.rerank(...) from cross_encoder.py to fill reranked[] and produce the top-N — the same RetrievalAudit contract Module C already emits, so no parallel data shape gets invented.

For the retriever backend, default to in_memory unless CRE_LIBRARIAN_RETRIEVER_BACKEND=pgvector is set and the DB is actually Postgres with embedding_vec populated (per the W3 loud-warning behavior) — reuses the existing config gate instead of adding a second one for MyOpenCRE.

Reuse get_embedding_contents_by_doc_type from db.py (added in #957) rather than writing a new embeddings-lookup query, and reuse the UUID→external_id translation pattern from scripts/evaluate_librarian.py/cre_main.py's resolve() fast path, since that's a real gotcha already hit twice in the Librarian PRs (recall read 0% until fixed).

Map RetrievalAudit.reranked[] scores to the user_standard_sections status enum with explicit thresholds documented in the RFC (proposing score_rerank >= 0.80 → AUTO_MAPPED, 0.50–0.80 → NEEDS_REVIEW, < 0.50 → UNMAPPED, informed by C.2's measured 75% top-1 accuracy at the 0.80 target) — happy to adjust these once I have a fixture set to validate against.

Keep the rq job wrapper in mapping_job.py unchanged; only its internal call swaps to the two Librarian calls above.
Tests: a fixture upload asserting deterministic status counts (relying on the stable-sort fix already landed in candidate_retriever.py/cross_encoder.py for tie-order determinism), plus a regression test asserting USER_CONFIRMED rows are excluded from the run_mapping write path on rerun.

One decision to confirm before I start: should the confidence thresholds live as constants in mapping_job.py itself, or in librarian/config_loader.py alongside CRE_LIBRARIAN_* so both MyOpenCRE and future Module C consumers share one source of truth? I'd lean toward the latter for consistency, but happy to scope it to mapping_job.py only if you'd rather keep MyOpenCRE's policy decisions separate from Module C's config surface.

Will keep the PR scoped to run_mapping + threshold mapping + tests per the acceptance criteria, no changes to C.3–C.5 or the pgvector migration itself.

[tanishbhandari11t]

@northdpole Hello, I would be interested in contributing towards this issue too. I have gone through the requirement for the same and the Module C retrieval/reranking flow as of now and find the idea of reuse to be especially fascinating.
From what I understand, @Glitch-On has already mentioned that he would like to contribute to this topic. However, if there are no further contributors and if the problem statement is still available, I would definitely like to contribute.

Thanks!