Repository navigation
test: intrinsic HF generation (adapter_scope routing) has no real-model e2e coverage #1574
Description
Epic: #929 | Blocked by: #1465 (PR #1555) | Phase: 2+
Problem
PR #1555 (#1465) routes LoRA/aLoRA intrinsic generation on LocalHFBackend through AdapterMixin.adapter_scope(): a new _IntrinsicPeftBinding driving load_peft_adapter/activate_peft_adapter/deactivate_peft_adapter, per-call _AdapterCore construction that reuses the resolved Identity, and the intrinsic path's first production firing of the ADAPTER_FUNCTION_* lifecycle hooks (activate/deactivate phase events plus invocation-complete carrying the catalogue-pinned revision and binding_type="local_file").
The new path is covered at in-CI granularity by MagicMock unit tests only (test_generate_intrinsic_with_adapter_scope_* in test/backends/test_huggingface_unit.py, including the error-outcome and prepare-failure payload pins added with the PR). They pin the plumbing precisely — verb order, lock state, hook payloads — but never exercise a real PeftModel, a real downloaded adapter, or the intrinsic prompt/parse round trip.
The existing GPU e2e (test/backends/test_adapters/test_local_file_e2e.py) proves the scope mechanism on a real model (the adapter stays active across a raw model.generate), but it drives LocalFileBinding (#1454), not the intrinsic path: _generate_intrinsic_with_adapter_scope itself has no real-model observation anywhere in the repo.
Net effect: the mid-pyramid rung is missing for this path — mock (in-CI) → gap → HPC qualitative runs (test_run_transformers and friends). A real-model regression in the verb wiring (PEFT load_adapter semantics, aLoRA activation behaviour, the load_peft_adapter already-loaded tolerance on a repeat call) would first surface in an HPC job, not a dev loop. Open question for scoping: if the HPC intrinsic suites run against this path on a regular cadence, the residual gap narrows to "faster local repro harness" — worth stating explicitly in the outcome comment of the #929 thread before this is claimed.
Expected
A structural, non-qualitative GPU e2e that drives one real intrinsic call end to end through the public API and asserts on the mechanism, not text content — per test/README.md e2e rules, and deliberately avoiding the score-content assertions that made GPU intrinsic tests flaky (#1291 documents exactly that failure class):
- Call — one intrinsic through the public surface (e.g.
core.check_certaintyon a minimalChatContext), a small real Granite model, real adapter download (catalogue-pinned SHA). - Hook payload — exactly one
ADAPTER_FUNCTION_INVOCATION_COMPLETEwithoutcome="success", correctname/adapter_type,binding_type="local_file", andrevisionequal to the catalogue-pinned SHA (the PR's labelling claim, verified on the real path), with phase events exactly["activate", "deactivate"]. Reusetest/backends/test_adapters/_hook_capture.py. - Real model state — post-call,
backend._model.active_adapters() == [](clean deactivation on a real PEFT model, not the hand-wired mock fake). - Repeat call — a second call for the same adapter proves the per-call
_IntrinsicPeftBindingis idempotent against a real model:load_peft_adapterexplicitly tolerates an already-loaded adapter (itsValueErrorcatch), a path the mocks structurally cannot exercise.
Out of scope: score/content-quality assertions (the qualitative/HPC lane's job); CI gating — this belongs in the hardware-gated e2e tier, like the existing GPU e2e.
Acceptance criteria
- New test in
test/backends/test_adapters/(existing e2e file if its fixture supports intrinsic adapters, else a sibling file), markede2e,huggingface,slow - All four assertions above pass against a real model on GPU (or equivalent local memory)
- The test has been re-run at least twice to shake out flakiness before being considered stable
- Model ID pinned in the test;
revisionasserted from the catalogue so drift fails loudly
Resources
GPU or sufficient local memory for a 3B-class Granite model, plus Hub access for the base model and intrinsic adapter download. Roughly a day including the first real-run debug cycle.
Activity
planetf1 commentedon Sep 2, 2026
Update (2026-09-02) — verified unaffected by #1018
#1018 (PR #1593, merged 2026-09-02) added
LocalHFBackendembedded-adaptersupport, but that's the
EmbeddedBinding/Granite Switch path — a differentbinding type from this issue's
LocalFileBinding/PEFT path(
_generate_intrinsic_with_adapter_scope,binding_type="local_file"). Noscope change. Confirmed the referenced files are still current:
test/backends/test_adapters/test_local_file_e2e.py,_hook_capture.py,and
mellea/stdlib/components/intrinsic/core.py::check_certaintyall existat HEAD. Dependency #1465 (PR #1555) is merged — ready to start now.
test(backends): add real-model GPU e2e for intrinsic adapter_scope ro…