Skip to content

Harden Servy V5 trained-model serving verification - #16

Merged
yashaskn8 merged 14 commits into
mainfrom
feature/servy-v5-serving-verification-20261025
Oct 29, 2026
Merged

yashaskn8 merged 14 commits into
mainfrom
feature/servy-v5-serving-verification-20261025

Conversation

@yashaskn8

@yashaskn8 yashaskn8 commented Oct 24, 2026 •

Copy link
Copy Markdown
Owner

Verified base

  • Based on main 358c48cad5301dc7dbd68f7f7b4fcecb530f512e.
  • V5 already has a local SHA-pinned checkpoint loader and an opt-in Django reranker; private weights are intentionally not stored in Git.

Change

  • Make python manage.py verify_v5_reranker run the same actual rerank_candidates path as production, including raw logits, instead of a separate default-activation model.predict() probe.
  • Treat disabled model, unavailable checkpoint, legacy fallback, and non-finite scoring as smoke-verification failures.
  • Add synthetic tests for raw-logit identity activation and failure handling; document scope and deployment instructions.

Important limitations

  • Not a production promotion. The actual 2.27 GB V5 best_model checkpoint and its running environment were not accessible, and an independent holdout is not established.
  • No model artifacts, personal/company text, secrets, or customer data were committed.
  • Baseline MiniLM remains default, and V5 is opt-in.
  • Real-weight path: offline installer -> SERVY_RERANKER_*=... -> smoke command -> SERVY_TEST_REAL_V5=1 end-to-end check -> independent quality/latency/citation/safety evaluation.
  • This PR does not claim the trained model is installed or deployed.

Verification

  • Awaiting GitHub MySQL backend tests, synthetic V5 tests, frontend lint/build. No real-weight test is runnable in CI absent the private checkpoint.

October 29 follow-up: required-model startup gate

  • Added opt-in SERVY_RERANKER_REQUIRED=1 with a synchronous Django CoreConfig.ready() check. In this mode the application refuses to start unless enabled, SHA-pinned V5 inference produces a finite score through the real reranker. Required-mode request-time inference failures also propagate instead of falling back to baseline, including through retrieve(). Default/off behavior stays unchanged.
  • Added eight synthetic regression tests for disabled/wrong-digest/missing-checkpoint conditions, successful raw-logit scoring, application startup failure, and subsequent request-time model failures; updated the environment example and V5 setup guide.
  • This is not proof of a loaded private checkpoint, production deployment, independent accuracy, or a successful real-weight HTTP request. SERVY_TEST_REAL_V5=1 remains an operator-run verification gate after approved weights are installed.
  • GitHub Actions for new head must be reviewed separately from prior CI runs.

@yashaskn8
yashaskn8 marked this pull request as ready for review October 29, 2026 07:47
@yashaskn8
yashaskn8 merged commit b602587 into main Oct 29, 2026
4 checks passed
@yashaskn8
yashaskn8 deleted the feature/servy-v5-serving-verification-20261025 branch October 29, 2026 09:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant