fix(backends): activate aLoRA adapters on the LocalHFBackend generate path by planetf1 · Pull Request #1685 · generative-computing/mellea

planetf1 · 2026-09-25T20:40:00Z

Pull Request

End user impact: every aLoRA adapter call through LocalHFBackend was
silently being answered by the base model. The adapter was downloaded,
loaded, and reported as active, but its weights were never applied to
generation, with no error, no warning, and nothing to notice in shape or
speed. That reached anyone who registered an aLoRA explicitly: a composed
Adapter with adapter_type="alora", or the deprecated IntrinsicAdapter
shim, which defaults to aLoRA. A default requirement_check() or
check_certainty() call was not affected, because resolve_adapter registers
the LoRA on this backend (mellea/backends/adapters/adapter.py:896-916); #1654
would make the aLoRA selectable there, which is when this becomes live for
everyone.

Issue

Related to #1679. Two independent bugs caused requirement-check aLoRA to
never activate:

Mellea-side activation bug. Fixed here.
Published io.yaml defect. The requirement-check aLoRA instruction
does not tokenise to the adapter's declared invocation sequence. The
adapter owners are correcting the io.yaml upstream, and chore(adapters): bump granitelib-core pin once requirement-check aLoRA io.yaml is republished #1699 tracks
picking that up (catalogue pin bump). Until then the requirement-check
aLoRA stays off on this backend and logs a WARNING; every other aLoRA
activates with this PR.

Root cause

An aLoRA adapter applies its weights only to tokens after an "invocation
sequence" (alora_invocation_tokens in the adapter config); earlier tokens
use the base model. PEFT computes the per-layer rank offsets from the prompt
and injects them only from inside its PeftModel wrapper's
generate()/forward() overrides. LocalHFBackend instead loads adapters
via model.load_adapter() on the bare model, so no wrapper exists, offsets
are never computed, and every aLoRA variant layer masks the adapter off.
Instrumentation of a full requirement-check run confirmed: zero PEFT
aLoRA hook calls.

How it was found. #1679 reported that requirement_check() returns
implausible scores. Running the failing example with logging on every place
PEFT could apply the aLoRA showed none of those places were ever reached:
the adapter was never applied during generation. A re-run with the fix
confirmed it — the same example flips from the base model's wrong answer to
the adapter's correct one.

Fix

generate_with_transformers(), the backend's
only generate call site that runs with an adapter active, now runs model.generate() inside
_alora_activation_context(). When exactly one active adapter is an aLoRA,
the context computes the offsets with PEFT's calculate_alora_offsets() and
injects them into every LoraLayer through temporary pre-forward hooks,
which is what PeftModel does internally. Passing alora_offsets= to
generate() is not enough on its own, because transformers does not forward
kwargs to the projection layers.

Edge cases follow PEFT: beam search raises a ValueError, offsets are
repeated for num_return_sequences > 1, and if the invocation sequence is
missing from the prompt the adapter stays off and a WARNING is logged once.

Scope. Only the bare model.load_adapter() path was affected.
m alora train and load_transformers_lora() wrap the model in PeftModel,
which already handles offsets, and Ollama, OpenAI, Watsonx, and LiteLLM apply
aLoRA server-side.

Why the existing tests missed it. They check for a valid 0–1 score,
which the base model also produces. The new regression test checks that the
offsets reach the aLoRA layer.

Impact (3b diagnostic eval, 140 items). The control arms (aLoRA loaded,
LoRA, nothing) score 110–112. With activation the score is 121 (McNemar
p ≈ 0.027), mostly from mechanical checks (word counts, list markup, language
ID), and both #1679 misses (French, bullets) flip to correct. The eval corrected
the requirement-check io.yaml by hand; with this PR alone that adapter still
cannot activate (see #1699). At 8b and 30b
the activated arms are flat against base, because the base model already
handles that probe set. That is a large part of why the bug went unnoticed.

Tests

Tests added to the respective file if code was changed
New code has 100% coverage if code was added
Ensure existing tests and github automation passes (a maintainer will kick off the github automation when the rest of the PR is populated)

New: test/formatters/granite/base/test_base_alora_activation.py

8 mechanism unit tests on a tiny random-weight LlamaForCausalLM
(no model download). The regression guard is
test_generate_with_transformers_uses_context: it fails when the
_alora_activation_context() wiring is removed.
test_no_offsets_reach_variant_without_context is a canary on upstream
PEFT behaviour (it passes either way). Also covered: correct offset
delivery, per-row offsets for num_return_sequences, the beam-search
refusal, the once-per-adapter missing-invocation warning, hook presence
inside the context and teardown on normal and error exits, and no-op
behaviour for no-adapter, plain-LoRA, and empty-invocation models.
2 differential e2e tests (qualitative, slow,
require_gpu(min_vram_gb=20); skipped on CI like the rest of the
huggingface e2e suite): with a real LocalHFBackend, the adapter-on vs
adapter-off score through the intrinsic path must move by more than 0.5.
One per affected capability, both on the as-published adapters:
uncertainty, and requirement-check. The requirement-check test is a strict
xfail until the corrected io.yaml is picked up (chore(adapters): bump granitelib-core pin once requirement-check aLoRA io.yaml is republished #1699); with
strict=True it reports XPASS as a failure on the first GPU run after
the fix arrives, so the marker gets removed rather than left behind.

Every new unit guard was checked against a mutation that removes its fix
(offset expansion, kwargs wiring, beam-search refusal, empty-invocation guard,
missing-invocation branch, context wiring); each mutation fails at least one
test.

Full fast suite (-m "not qualitative"): 4563 passed, 0 failed.

Behaviour change to note. With activation working, every aLoRA adapter
on LocalHFBackend now actually applies its weights: for example
answerability, which the IntrinsicAdapter shim registers as aLoRA by
default, and the four guardian aLoRAs (guardian-core, policy-guardrails,
factuality-detection, and factuality-correction). Only requirement-check and
uncertainty were compared against a control arm here. Beam search with an
active aLoRA now raises ValueError, where it used to run silently on the
base model. test/backends/test_huggingface.py, which registers shim
aLoRAs and is skipped on CI, passes locally (28/28) with this change.

Test results

Unit (mechanism, 8): pass locally, no model download.
e2e (differential, 2): run locally on granite-4.1-3b with -m slow:
uncertainty passes (adapter-on/off gap 0.67–0.87); requirement-check
reports XFAIL, as expected on the as-published io.yaml. With the colon
removed (a load-time repair in an earlier revision of this PR, since
dropped), the same item scored adapter-on ≈ 0.05 vs
adapter-off ≈ 0.999, matching the diagnostic harness probe (0.0474 vs
0.9999).
mypy and pre-commit clean.

Other changes

None.

… path aLoRA intrinsic functions called through LocalHFBackend silently ran the base model: PEFT computes and injects alora_offsets only from inside its PeftModel wrapper's generate()/forward() overrides, and the backend loads adapters via the bare model.load_adapter() path, so the aLoRA variant layers never received offsets and masked the adapter off for every token. generate_with_transformers() now wraps the model.generate() call in _alora_activation_context(): a no-op unless exactly one active adapter declares alora_invocation_tokens, in which case it computes offsets with PEFT's public calculate_alora_offsets and registers PEFT's own 2-line pre-forward hooks on every LoraLayer for the duration of the call. The hook is required because transformers' model code does not propagate kwargs to the projection Linear calls. Handles both bare PeftAdapterMixin models (active_adapters as method) and PeftModel wrappers (property; idempotent there). Also repairs, with a warning, the published requirement-check io.yaml instruction that does not tokenise to its own declared invocation sequence (issue generative-computing#1679): verification-driven and self-terminating, so a correctly republished adapter file is left untouched in either fix direction. The repair block is intentionally removable if we prefer Mellea not to touch published prompt text. Tests: 8 mechanism unit tests on a tiny random model (no download), 8 repair unit tests on a fake tokenizer mimicking the Granite BPE merge of the '>' and ':' tokens, and 2 qualitative e2e differential tests (requirement-check on the as-published adapter, uncertainty) asserting adapter-on vs adapter-off moves the score by more than 0.5 through the real backend path. Assisted-by: pi Signed-off-by: Nigel Jones <jonesn@uk.ibm.com>

…ctivation edges Review follow-ups for the LocalHFBackend aLoRA activation fix (generative-computing#1679). - Apply the io.yaml invocation repair on the deprecated IntrinsicAdapter path too. That shim loads its weights per generate call, after its config has been rendered into the prompt, so the declared sequence is read from the downloaded adapter_config.json at registration. The repair now returns a copy instead of mutating, since a shim's config can be the caller's dict. - Refuse beam search for an activating aLoRA, as PEFT's own path does, and repeat offsets per row for num_return_sequences > 1; both previously failed with an opaque IndexError inside the variant layer. - When the declared invocation sequence is absent from the prompt, register no hooks and log a WARNING once per adapter instead of staying silent. - Treat an empty alora_invocation_tokens as a no-op, and tolerate only the "No adapter loaded" ValueError from active_adapters(). - Tighten types, strengthen the no-op and teardown tests to assert hook counts inside the context, relabel the upstream-canary test, add require_gpu to the e2e classes, and drop their dead gh_run xfail. Assisted-by: Claude Code Signed-off-by: Nigel Jones <jonesn@uk.ibm.com>

Drop unit tests that duplicate another test or pin a trivial early return, keeping every guard that catches a regression in the fix: - activation: remove the context-level num_return_sequences test (the production-path test covers the same expansion), merge the two hook teardown tests and the three no-op tests, and drop the ValueError narrowing test. - repair: remove the direct tests of the small helpers and of the method's early returns, which the repair-logic and add_adapter wiring tests already exercise; reuse test_huggingface_unit's _make_backend instead of a copy, and assert the repair warning in the shim wiring test. Assisted-by: Claude Code Signed-off-by: Nigel Jones <jonesn@uk.ibm.com>

jakelorocco

approving because it lgtm; a few nits

jakelorocco · 2026-09-28T20:25:12Z

+    def _repair_alora_instruction(
+        self,
+        io_yaml_config: dict,
+        invocation_tokens: Sequence[int] | None,
+        adapter_name: str,
+    ) -> dict:
+        """Repair an aLoRA io.yaml instruction that cannot activate its own adapter.
+
+        Some published adapters (issue #1679: `requirement-check`, all
+        granite-4.1 slots) ship an instruction whose text does not tokenise to
+        the `alora_invocation_tokens` declared in the adapter's own config,
+        so the adapter can never activate no matter what the loading path
+        does. This loads a locally repaired instruction instead, with a loud
+        warning, so the capability works end to end until the publisher
+        republishes a corrected file. The repair itself is
+        verification-driven and self-terminating (see
+        `_alora_invocation_repair`): a healthy file -- in either direction a
+        publisher might fix it -- is returned untouched.
+
+        Called from both `add_adapter` registration paths: the composed-adapter
+        commit point, with the declared sequence read from the PEFT config
+        `binding.prepare()` loaded, and the `IntrinsicAdapter` shim path, with
+        it read from the downloaded `adapter_config.json` (that shim loads its
+        weights per generate call, after its config has already been rendered
+        into the prompt).
+
+        Args:
+            io_yaml_config: The adapter's `io.yaml` mapping. Never mutated: a
+                shim's config can be the caller's own `config_dict`.
+            invocation_tokens: The adapter's declared
+                `alora_invocation_tokens`, or `None` for a non-aLoRA adapter.
+            adapter_name: The adapter's qualified name, for the warning.
+
+        Returns:
+            `io_yaml_config` itself when no repair applies, otherwise a shallow
+            copy carrying the repaired instruction.
+        """
+        if not invocation_tokens:
+            return io_yaml_config
+        instruction = io_yaml_config.get("instruction")
+        if not isinstance(instruction, str) or not instruction:
+            return io_yaml_config
+        repaired = _alora_invocation_repair(
+            self._tokenizer, instruction, invocation_tokens
+        )
+        if repaired is None:
+            return io_yaml_config
+        MelleaLogger.get_logger().warning(
+            f"Adapter {adapter_name!r}: the published io.yaml instruction does "
+            "not tokenise to the adapter's declared aLoRA invocation sequence, so "
+            "the adapter could never activate as published. Loaded a locally "
+            "repaired instruction instead (issue #1679). Ask the adapter "
+            "publisher to republish a corrected io.yaml."
+        )
+        return {**io_yaml_config, "instruction": repaired}


I will reach out to the granite library team and see if we can get them to fix this on their end. We shouldn't have to have these patches on our side.

Yep - seen the messages. Let's hold off from merging until we get a reply. I included the workaround to show one approach (and ensure full testing) but we can remove if a timely fix is available

Having thought again, I’ve pushed a new commit which removes the workaround. The only concession is to make the test case which would fail now do an xfail instead

If #1654 is merged before the io.yaml fix we don’t get any failures as such, though requirement_check() on LocalHFBackend will get worse in quality.

Since I’m hopeful we’ll get an io.yaml update soon this seems better than including the workaround code — I’ve kept that on another branch (and it’s in commit history there) in case we end up deciding to add it in this pr or a followup.

jakelorocco · 2026-09-28T20:27:09Z

        return
+
+
+def _token_sequence_present(


Maybe we should start moving some of these adapter helper functions to a separate file?

@jakelorocco

…arch The aLoRA activation context read num_beams and num_return_sequences only from flat generate() kwargs and the model default, so a caller passing generation_config=GenerationConfig(num_beams=...) through model_options bypassed the beam-search refusal and the per-row offset expansion, and hit an opaque IndexError in the variant layer. Resolve settings in generate()'s own order: flat kwarg, then a generation_config kwarg, then the model default (unset GenerationConfig fields are None, so each level falls through). Reported in review by @jakelorocco, whose tests are added as generation-config cases of the existing beam-search and multi-sequence tests. Assisted-by: Claude Code Signed-off-by: Nigel Jones <jonesn@uk.ibm.com>

@jakelorocco

…own module _token_sequence_present, _read_alora_invocation_tokens and _alora_invocation_repair have no dependency on the backend, so move them from huggingface.py into mellea/backends/adapters/_alora_repair.py. The module docstring records that it is the removable generative-computing#1679 workaround, so retiring it is a one-file delete plus LocalHFBackend._repair_alora_instruction and its two call sites. Suggested in review by @jakelorocco. Assisted-by: Claude Code Signed-off-by: Nigel Jones <jonesn@uk.ibm.com>

…hten tests Second-round review follow-ups for the generative-computing#1679 aLoRA work. - Apply the composed-Adapter io.yaml repair before binding.prepare(), reading the declared invocation sequence from the downloaded adapter_config.json as the IntrinsicAdapter path already does. The post-prepare commit block goes back to its two dict assignments, so a failure in the repair leaves nothing registered. - Document the ValueError that generate_with_transformers can now raise for beam search with an active aLoRA, and list the attributes the logits capture proxy must now forward. - Make the fake tokenizer reproduce the real three-token invocation ([<, requirements, >]) and the >: merge, so the multi-token match is exercised without a model download. - Move the repair tests to test/backends/test_adapters/test_alora_repair.py to mirror the new module, and build the e2e backends through a fixture that uses hf_skip() and cleanup_gpu_backend(). Assisted-by: Claude Code Signed-off-by: Nigel Jones <jonesn@uk.ibm.com>

planetf1 · 2026-09-29T10:44:24Z

Moved to DRAFT whilst awaiting for a reply from the granite team & deciding whether we merge in the workaround or wait for upstream fix.

The published requirement-check aLoRA io.yaml is being corrected upstream, so remove the load-time instruction repair rather than carry it in Mellea. The activation fix is unaffected: an adapter whose invocation sequence is missing from the prompt stays off and logs a warning. Mark the requirement-check differential e2e test as a strict xfail until the adapter is republished and the catalogue pin is bumped. Assisted-by: Claude Code Signed-off-by: Nigel Jones <jonesn@uk.ibm.com>

github-actions Bot added the bug Something isn't working label Sep 25, 2026

This was referenced Sep 25, 2026

bug(adapters): published requirement-check aLoRA never activates, and scores worse than the LoRA #1679

Closed

feat(adapters)!: prompt fallback for adapter functions without a trained adapter #1654

Open

planetf1 marked this pull request as ready for review September 26, 2026 20:40

planetf1 requested a review from a team as a code owner September 26, 2026 20:40

planetf1 requested review from ajbozarth, jakelorocco and markstur September 26, 2026 20:40

planetf1 closed this Sep 26, 2026

planetf1 reopened this Sep 26, 2026

planetf1 added 2 commits September 28, 2026 10:09

jakelorocco approved these changes Sep 28, 2026

View reviewed changes

planetf1 added 3 commits September 29, 2026 09:50

planetf1 mentioned this pull request Sep 29, 2026

Ollama removed LoRA-adapter GGUF *creation* (v0.34.1+) — breaks adapter provisioning for #1634, serving unaffected #1651

Open

planetf1 marked this pull request as draft September 29, 2026 10:43

planetf1 mentioned this pull request Sep 30, 2026

chore(adapters): bump granitelib-core pin once requirement-check aLoRA io.yaml is republished #1699

Closed

4 tasks

jakelorocco approved these changes Sep 30, 2026

View reviewed changes

planetf1 marked this pull request as ready for review September 30, 2026 18:40

planetf1 added this pull request to the merge queue Sep 30, 2026

Merged via the queue into generative-computing:main with commit a5a10b2 Sep 30, 2026
14 checks passed

planetf1 deleted the issue-1679 branch September 30, 2026 19:12

planetf1 mentioned this pull request Oct 5, 2026

chore(adapters): bump granitelib-core pin for fixed requirement-check io.yaml #1706

Merged

Navigation Menu

Repository navigation

fix(backends): activate aLoRA adapters on the LocalHFBackend generate path - #1685

fix(backends): activate aLoRA adapters on the LocalHFBackend generate path#1685
planetf1 merged 7 commits into
generative-computing:mainfrom
planetf1:issue-1679

planetf1 commented Sep 25, 2026 •

edited

Loading

Uh oh!

jakelorocco left a comment

Uh oh!

jakelorocco Sep 28, 2026

Uh oh!

planetf1 Sep 29, 2026

Uh oh!

planetf1 Sep 30, 2026

Uh oh!

jakelorocco Sep 28, 2026

Uh oh!

planetf1 Sep 30, 2026

Uh oh!

Uh oh!

planetf1 commented Sep 29, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

planetf1 commented Sep 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Pull Request

Issue

Root cause

Fix

Tests

Test results

Other changes

Uh oh!

jakelorocco left a comment

Choose a reason for hiding this comment

Uh oh!

jakelorocco Sep 28, 2026

Choose a reason for hiding this comment

Uh oh!

planetf1 Sep 29, 2026

Choose a reason for hiding this comment

Uh oh!

planetf1 Sep 30, 2026

Choose a reason for hiding this comment

Uh oh!

jakelorocco Sep 28, 2026

Choose a reason for hiding this comment

Uh oh!

planetf1 Sep 30, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

planetf1 commented Sep 29, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

planetf1 commented Sep 25, 2026 •

edited

Loading