Skip to content

test: intrinsic HF generation (adapter_scope routing) has no real-model e2e coverage #1574

Closed

Description

@planetf1

Epic: #929 | Blocked by: #1465 (PR #1555) | Phase: 2+

Problem

PR #1555 (#1465) routes LoRA/aLoRA intrinsic generation on LocalHFBackend through AdapterMixin.adapter_scope(): a new _IntrinsicPeftBinding driving load_peft_adapter/activate_peft_adapter/deactivate_peft_adapter, per-call _AdapterCore construction that reuses the resolved Identity, and the intrinsic path's first production firing of the ADAPTER_FUNCTION_* lifecycle hooks (activate/deactivate phase events plus invocation-complete carrying the catalogue-pinned revision and binding_type="local_file").

The new path is covered at in-CI granularity by MagicMock unit tests only (test_generate_intrinsic_with_adapter_scope_* in test/backends/test_huggingface_unit.py, including the error-outcome and prepare-failure payload pins added with the PR). They pin the plumbing precisely — verb order, lock state, hook payloads — but never exercise a real PeftModel, a real downloaded adapter, or the intrinsic prompt/parse round trip.

The existing GPU e2e (test/backends/test_adapters/test_local_file_e2e.py) proves the scope mechanism on a real model (the adapter stays active across a raw model.generate), but it drives LocalFileBinding (#1454), not the intrinsic path: _generate_intrinsic_with_adapter_scope itself has no real-model observation anywhere in the repo.

Net effect: the mid-pyramid rung is missing for this path — mock (in-CI) → gap → HPC qualitative runs (test_run_transformers and friends). A real-model regression in the verb wiring (PEFT load_adapter semantics, aLoRA activation behaviour, the load_peft_adapter already-loaded tolerance on a repeat call) would first surface in an HPC job, not a dev loop. Open question for scoping: if the HPC intrinsic suites run against this path on a regular cadence, the residual gap narrows to "faster local repro harness" — worth stating explicitly in the outcome comment of the #929 thread before this is claimed.

Expected

A structural, non-qualitative GPU e2e that drives one real intrinsic call end to end through the public API and asserts on the mechanism, not text content — per test/README.md e2e rules, and deliberately avoiding the score-content assertions that made GPU intrinsic tests flaky (#1291 documents exactly that failure class):

  1. Call — one intrinsic through the public surface (e.g. core.check_certainty on a minimal ChatContext), a small real Granite model, real adapter download (catalogue-pinned SHA).
  2. Hook payload — exactly one ADAPTER_FUNCTION_INVOCATION_COMPLETE with outcome="success", correct name/adapter_type, binding_type="local_file", and revision equal to the catalogue-pinned SHA (the PR's labelling claim, verified on the real path), with phase events exactly ["activate", "deactivate"]. Reuse test/backends/test_adapters/_hook_capture.py.
  3. Real model state — post-call, backend._model.active_adapters() == [] (clean deactivation on a real PEFT model, not the hand-wired mock fake).
  4. Repeat call — a second call for the same adapter proves the per-call _IntrinsicPeftBinding is idempotent against a real model: load_peft_adapter explicitly tolerates an already-loaded adapter (its ValueError catch), a path the mocks structurally cannot exercise.

Out of scope: score/content-quality assertions (the qualitative/HPC lane's job); CI gating — this belongs in the hardware-gated e2e tier, like the existing GPU e2e.

Acceptance criteria

  • New test in test/backends/test_adapters/ (existing e2e file if its fixture supports intrinsic adapters, else a sibling file), marked e2e, huggingface, slow
  • All four assertions above pass against a real model on GPU (or equivalent local memory)
  • The test has been re-run at least twice to shake out flakiness before being considered stable
  • Model ID pinned in the test; revision asserted from the catalogue so drift fails loudly

Resources

GPU or sufficient local memory for a 3B-class Granite model, plus Hub access for the base model and intrinsic adapter download. Roughly a day including the first real-run debug cycle.

Activity

  1. self-assigned this
    on Aug 20, 2026
  2. added
    area/backendsProvider-specific work: Ollama, HF, LiteLLM, OpenAI, Bedrock, vLLM
    area/adapter-functionsGranite adapter functions: framework and adaptiers including RAG, Guardian, Core
    on Aug 20, 2026
  3. planetf1 commented on Sep 2, 2026

    @planetf1
    ContributorAuthor

    Update (2026-09-02) — verified unaffected by #1018

    #1018 (PR #1593, merged 2026-09-02) added LocalHFBackend embedded-adapter
    support, but that's the EmbeddedBinding/Granite Switch path — a different
    binding type from this issue's LocalFileBinding/PEFT path
    (_generate_intrinsic_with_adapter_scope, binding_type="local_file"). No
    scope change. Confirmed the referenced files are still current:
    test/backends/test_adapters/test_local_file_e2e.py, _hook_capture.py,
    and mellea/stdlib/components/intrinsic/core.py::check_certainty all exist
    at HEAD. Dependency #1465 (PR #1555) is merged — ready to start now.

  4. added a commit that references this issue on Sep 3, 2026
    8359182
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

area/adapter-functionsGranite adapter functions: framework and adaptiers including RAG, Guardian, Corearea/backendsProvider-specific work: Ollama, HF, LiteLLM, OpenAI, Bedrock, vLLM

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions