Skip to content

bug(adapters): detect when an aLoRA cannot activate, and stop selecting it silently #1678

Closed

Description

@planetf1

What happens today

You call an adapter function on a backend that loads adapters locally, say core.requirement_check(...).
Mellea downloads an aLoRA, loads it, and reports it as active. You get a score back that looks
perfectly normal.

If the adapter's activation point never appears in the prompt Mellea built, the adapter never switches
on. It contributes nothing it was trained to contribute. Your score came from the base model instead.
Nothing in the returned value, the logs, or the telemetry says so. list_adapters() shows the adapter.
active_adapters() on the model shows the adapter. Every signal says it ran.

Why that is a problem

An aLoRA only applies from the point where a declared token sequence appears in the input. PEFT
implements this, and when the sequence is missing it gives up quietly:

# peft/tuners/lora/variants.py, calculate_alora_offsets()
            else:  # Invocation sequence not found in input
                alora_offsets[i] = None

There is no warning and no error. So a user gets a number with no way to know it is worthless, and we
have no way to know either. This is worse than a failure, because the value is well formed and passes
every schema check we apply to it.

We have one confirmed case in the published adapters right now. The requirement-check aLoRA declares
an activation sequence its own instruction text cannot produce, so on Granite 4.1 models it never
activates. Details are in the linked upstream issue.

How it should behave

When Mellea loads an aLoRA and the declared activation sequence is absent from the prompt it assembled,
one of two things should happen:

  • the situation is reported clearly, naming the capability, the base model, and the decoded sequence, or
  • the adapter is treated as unavailable, so resolution falls through to the next rung and the user gets
    a working adapter instead.

Either way it stops being invisible.

Where the check belongs

Mellea is the only component that holds both halves at once. It has the loaded adapter's config and the
prompt it just built. PEFT sees the mismatch and says nothing, the base model cannot know, and the user
has no access to either. The check itself is a subsequence search over token ids we already have.

Detail: the check has to run against the assembled prompt, not the instruction

Not every aLoRA activates on text from its own io.yaml. Of the five aLoRA-backed capabilities
published for granite-4.1-3b:

  • uncertainty declares <certainty>, which its instruction contains.
  • answerability, query_rewrite and query_clarification declare
    <|start_of_role|>assistant<|end_of_role|>, which the chat template adds.
  • requirement-check declares <requirements>, which its instruction defeats through tokenisation.

A test of the form "the sequence must appear in the instruction" would wrongly reject three of the five.

Detail: possible shapes
  1. Generation-time guard. After building input_ids, if the active adapter declares
    alora_invocation_tokens and the sequence is absent, log a warning. Cheapest option, covers
    template-driven activation, and turns a silent wrong answer into something diagnosable.
  2. Resolution-time gate. Treat an aLoRA that cannot activate as unavailable so the walk selects the
    LoRA. This is what "prefer aLoRA where the infrastructure supports it" means in practice. Needs care,
    because at resolve time the chat template has not been applied yet.
  3. Both, with the gate deciding selection and the guard catching whatever the gate cannot predict.

Backends that load PEFT adapters all inherit the problem. For server-mediated and embedded deployments
activation happens outside Mellea, so those paths need answering separately rather than assuming.

Acceptance criteria

  1. An aLoRA whose declared activation sequence is absent from the assembled prompt is either reported
    clearly or treated as unavailable, on every backend that loads adapters itself.
  2. Tests cover both arms: a sequence that is present activates and stays quiet, a sequence that is
    absent is reported or skipped.
  3. A pin bump that promotes an aLoRA rung carries evidence that the aLoRA is no worse than the LoRA on
    accuracy and on false-pass rate for that capability. The catalogue pins commit SHAs
    (_CORE_R1_SHA and siblings in mellea/backends/adapters/catalog.py), so an upstream fix only
    reaches users when somebody bumps a pin. That bump is the moment an adapter stops being skipped and
    becomes the default path, which makes it the right place to ask for evidence.
  4. A comment beside the pinned SHAs in catalog.py names that requirement, so whoever bumps them finds
    it where they are working.
  5. This issue owns the flip back to aLoRA-first. Following the review of feat(adapters)!: prompt fallback for adapter functions without a trained adapter #1654, that PR keeps today's
    trained-rung order and probes LoRA before aLoRA, so promotion no longer happens through a pin bump: it
    happens by reordering the probe in _register_available_composed_adapter. Once the guard exists and the
    evidence question in bug(adapters): published requirement-check aLoRA never activates, and scores worse than the LoRA #1679 is settled, reorder it here. Criterion 3 stays as the trigger for any other
    capability whose aLoRA arrives through a catalogue bump rather than through this reorder.

Related

Found while reviewing #1654, which makes the aLoRA rung selectable on the local Hugging Face path for
the first time. The published artefact problem that motivates criteria 3 and 4 is tracked in
#1679. Downstream visibility of a changed mechanism is #1680. The equivalent
question for server-mediated adapters is noted on #1651.

Activity

  1. added
    bugSomething isn't working
    area/aloraLoRA/aLora adapter training
    on Sep 25, 2026
  2. self-assigned this
    on Sep 25, 2026
  3. added
    area/adapter-functionsGranite adapter functions: framework and adaptiers including RAG, Guardian, Core
    on Sep 25, 2026
  4. changed the title [-]bug(adapters): an aLoRA that cannot activate is used silently, with no signal to the user[/-] [+]bug(adapters): detect when an aLoRA cannot activate, and stop selecting it silently[/+] on Sep 25, 2026
  5. planetf1 commented on Sep 25, 2026

    @planetf1
    ContributorAuthor

    building a fix

  6. added a commit that references this issue on Oct 2, 2026
    4c9a3c1
  7. added 3 commits that reference this issue on Oct 5, 2026
    51cbdb5
    7a9536f
    587d57c
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

area/adapter-functionsGranite adapter functions: framework and adaptiers including RAG, Guardian, Corearea/aloraLoRA/aLora adapter trainingbugSomething isn't working

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions