Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -245,6 +245,7 @@ post-activation shape each produces:
| `LocalFileBinding` | LocalFile/PEFT | `prepare` / `activate` / `deactivate` / `release` | `activate()` / `deactivate()`, via `adapter_scope` | Backend-internal PEFT adapter state toggled; the outgoing request is untouched |
| `EmbeddedBinding` | Embedded/Granite Switch | none — weights are already in the served model | `apply_activation(request, identity)` | `request.extra_body["chat_template_kwargs"]["adapter_name"]` set; `request.api_params["model"]` removed if present |
| `ServerMediatedBinding` | Ollama bundled adapter model | none for the current Ollama path | select the configured model tag during intrinsic generation | Ollama request's `model` is the bundled adapter tag; full lifecycle telemetry remains follow-up work |
| `PromptBinding` | Prompt fallback (weightless) | `prepare` / `activate` / `deactivate` / `release` — all no-ops except `activate()`, which logs | `activate()`, via the backend's weightless generate path (no `adapter_scope`) | None — generation runs against the base model with the rewriter-injected instruction |

### Project Resources

Expand Down
54 changes: 43 additions & 11 deletions docs/docs/advanced/intrinsics.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,10 @@ LoRA/aLoRA weights are loaded locally, selected from an embedded checkpoint, or
bundled into an Ollama model — faster and more reliable than prompting a
general-purpose model for these specialised micro-tasks.

When no trained adapter exists for a capability on the target model, resolution
transparently falls back to a hand-authored prompt run against the base model, so the
capability stays available — see [Prompt fallback](#prompt-fallback).

> **Backend note:** Adapter functions work with three backends:
>
> - **LocalHFBackend** — loads LoRA/aLoRA adapters from the catalog at runtime.
Expand Down Expand Up @@ -341,18 +345,40 @@ For a fully custom, non-catalog adapter — your own trained LoRA/aLoRA weights,
not one of the built-in adapter functions — see
[Adding a custom adapter function in 20 lines](../tutorials/07-custom-adapter-function.md).

## Prompt fallback

Not every capability has a trained adapter for every model. When no aLoRA or LoRA
adapter exists for a capability on the target model, resolution falls back to a
**prompt**: a hand-authored instruction (packaged as an `io.yaml`) run against the
plain base model. This keeps the capability available — including on models a LoRA
cannot be loaded into — rather than failing.

Resolution walks the chain in order and uses the first rung that exists:

1. a model-specific aLoRA adapter,
2. a model-specific LoRA adapter,
3. a model-specific prompt,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Two additions worth making here, both about what a user can tell from a result.

The section says the fallback is transparent, and flags reduced accuracy only for the any rung. A model-specific prompt also differs in quality from trained weights, and for a scoring helper that is the difference between a calibrated number and the model's self-report. One sentence would set expectations honestly.

There is also no documented way to find out which rung served a call. The value carries no provenance, and the telemetry route needs the hooks extra plus a plugin. backend.resolve_adapter(name).weights.binding_type already answers it, is public, and appears nowhere in the docs. Worth a line. Result-level provenance for the harder cases is #1680.

4. a model-agnostic `any` prompt (Mellea logs a warning, since an `any` prompt is
less accurate than a model-tuned one).

The fallback is transparent: the same helper calls (`rag.check_answerability`,
`core.requirement_check`, …) and `Intrinsic` components work unchanged — there is no
adapter to register and nothing to activate. A prompt fallback carries no weights (it
is a `PromptBinding`; see below) and runs the intrinsic's instruction against the base
model, parsing the response through the same output contract a trained adapter uses.

## Composable adapter construction (advanced)

`Adapter` composes an `Identity`, an output contract (`IOContract`), and a
weights binding into a single, inspectable object. Both `LocalHFBackend` and
`OpenAIBackend` accept a composed `Adapter` directly via `add_adapter` —
dispatching on the weights binding's reality (`LocalFileBinding` for
LocalFile/PEFT, `EmbeddedBinding` for Embedded/Granite Switch) — alongside
the deprecated shim classes, which remain functional for now (Epic #929,
issue #1144). An `EmbeddedBinding` adapter additionally requires `add_adapter`'s
`config=` argument (the raw io.yaml mapping) — `add_adapter` raises `ValueError`
without it, since that reality's config cannot be cheaply re-derived later; see
below.
LocalFile/PEFT, `EmbeddedBinding` for Embedded/Granite Switch, `PromptBinding`
for a weightless prompt fallback) — alongside the deprecated shim classes,
which remain functional for now (Epic #929, issue #1144). An `EmbeddedBinding`
adapter additionally requires `add_adapter`'s `config=` argument (the raw
io.yaml mapping) — `add_adapter` raises `ValueError` without it, since that
reality's config cannot be cheaply re-derived later; see below.

Each weights binding models how its deployment turns an adapter on.
`LocalFileBinding` downloads and loads LoRA/aLoRA weights, so it exposes a
Expand Down Expand Up @@ -401,13 +427,19 @@ switch_backend.register_embedded_adapter_model(
)
```

`PromptBinding` has no weights to manage, so there is nothing to load or activate.
Its lifecycle verbs are no-ops (`activate()` only logs that the fallback fired), and
generation runs against the base model with the injected instruction. You never
construct one directly; `resolve_adapter` registers it when the chain lands on a
prompt.

Weights-binding support by backend today:

| Backend | `LocalFileBinding` (LocalFile/PEFT) | `EmbeddedBinding` (Embedded/Granite Switch) | `ServerMediatedBinding` |
| --- | --- | --- | --- |
| `LocalHFBackend` | ✅ shipping — `add_adapter` accepts a composed `Adapter` or a bare `LocalFileBinding` directly | ✅ shipping — `load_embedded_adapters=True`, or `add_adapter(adapter, config=...)`/`register_embedded_adapter_model` with a composed `Adapter` | — |
| `OpenAIBackend` | — | ✅ shipping — `load_embedded_adapters=True`, or `add_adapter(adapter, config=...)`/`register_embedded_adapter_model` with a composed `Adapter` | — |
| `OllamaModelBackend` | — | — | ✅ model selection through `adapter_models` for catalogued adapter functions; lifecycle telemetry is tracked separately |
| Backend | `LocalFileBinding` (LocalFile/PEFT) | `EmbeddedBinding` (Embedded/Granite Switch) | `ServerMediatedBinding` | `PromptBinding` (prompt fallback) |
| --- | --- | --- | --- | --- |
| `LocalHFBackend` | ✅ shipping — `add_adapter` accepts a composed `Adapter` or a bare `LocalFileBinding` directly | ✅ shipping — `load_embedded_adapters=True`, or `add_adapter(adapter, config=...)`/`register_embedded_adapter_model` with a composed `Adapter` | — | ✅ shipping — registered by `resolve_adapter` when the fallback chain lands on a prompt |
| `OpenAIBackend` | — | ✅ shipping — `load_embedded_adapters=True`, or `add_adapter(adapter, config=...)`/`register_embedded_adapter_model` with a composed `Adapter` | — | — |
| `OllamaModelBackend` | — | — | ✅ model selection through `adapter_models` for catalogued adapter functions; lifecycle telemetry is tracked separately | — |

`ServerMediatedBinding` currently supports Ollama's bundled-model path. A full
server-mediated lifecycle and telemetry contract is tracked separately.
Expand Down
11 changes: 6 additions & 5 deletions docs/docs/advanced/lora-and-alora-adapters.md
Original file line number Diff line number Diff line change
Expand Up @@ -187,11 +187,12 @@ adapter is preferred whenever one is loaded, with three exceptions:
2. The requirement uses the `LLMaJRequirement` subtype explicitly — the caller is
asking for LLM-as-a-judge regardless of what adapters are loaded.
3. The adapter is unavailable (e.g. cannot be loaded) — Mellea falls back to
LLM-as-a-judge automatically. This is the *only* fallback case: if the
adapter runs but its output fails schema validation, `validate()` does not
fall back. Instead it surfaces the schema error on `ValidationResult.error`
and fails the check closed (`bool(result)` is `False`), so callers can tell
an unparsable adapter response apart from an ordinary "requirement not met".
LLM-as-a-judge automatically. This is the *only* case where the adapter path

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

While this paragraph is being edited, the model it describes has moved on.

It says routing follows from an adapter being loaded via backend.add_adapter(), which reads as opt-in by the user. After this PR, any adapter-function call resolves and registers an adapter implicitly, and on Granite 4.1 models that adapter can be an aLoRA, at which point the reroute gate matches and plain Requirement.validate() stops using LLM-as-a-judge. Same user code, different scoring mechanism, depending on the base model and on whether an adapter-function call happened earlier in the session.

Two things would help: say that resolution now registers adapters implicitly, and name default_to_constraint_checking_alora=False as the escape hatch for anyone who wants LLM-as-a-judge back. The PR body's "behaviour change worth a close look" paragraph covers the adapter-type switch, but not this consequence of it.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One more gap for this paragraph: on a model that only has a prompt fallback (no trained requirement-check adapter), Requirement.validate() still uses LLM-as-a-judge. The reroute gate only looks for "alora" adapters (huggingface.py:724-729), but the prompt adapter is registered as type "prompt" (adapter.py:597), so the gate never finds it.

Not a regression — those models had no adapter at all before this PR — but the docs should say the fallback doesn't cover the requirement path: the intrinsic helpers run the prompt while validate() runs LLMaJ on the same model. To get the prompt, ALoraRequirement(..., adapter_types=(AdapterType.PROMPT,)) works — the gate honours an explicit adapter_types (huggingface.py:723).

falls back to LLM-as-a-judge: if the adapter runs but its output fails schema
validation, `validate()` does not fall back. Instead it surfaces the schema
error on `ValidationResult.error` and fails the check closed (`bool(result)`
is `False`), so callers can tell an unparsable adapter response apart from an
ordinary "requirement not met".

If you want to force the adapter path even when using `generate_from_context`
directly (bypassing the normal `validate()` call), use `ALoraRequirement` from
Expand Down
2 changes: 2 additions & 0 deletions mellea/backends/adapters/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@
Identity,
IOContract,
LocalFileBinding,
PromptBinding,
ServerMediatedBinding,
WeightsBinding,
)
Expand Down Expand Up @@ -43,6 +44,7 @@
"IntrinsicAdapter",
"LocalFileBinding",
"LocalHFAdapter",
"PromptBinding",
"ServerMediatedBinding",
"WeightsBinding",
"fetch_intrinsic_metadata",
Expand Down
131 changes: 113 additions & 18 deletions mellea/backends/adapters/_core.py
Original file line number Diff line number Diff line change
Expand Up @@ -104,22 +104,23 @@ class Identity:

Attributes:
name (str): Human-readable adapter name.
adapter_type (Literal["lora", "alora"]): The LoRA variant.
adapter_type (Literal["lora", "alora", "prompt"]): The adapter variant. `"prompt"` is the
weightless prompt fallback.
capability (str | None): Advisory capability string; emits
:class:`UserWarning` when not in
:data:`~mellea.backends.adapters.capabilities.KNOWN_CAPABILITIES`.
"""

name: str
adapter_type: Literal["lora", "alora"]
adapter_type: Literal["lora", "alora", "prompt"]
capability: str | None = None

def __post_init__(self) -> None:
# Literal[...] is a static-only constraint; mypy enforces it but Python
# does not, so validate at runtime too.
if self.adapter_type not in ("lora", "alora"):
if self.adapter_type not in ("lora", "alora", "prompt"):
raise ValueError(
f"adapter_type must be 'lora' or 'alora', got {self.adapter_type!r}"
f"adapter_type must be 'lora', 'alora', or 'prompt', got {self.adapter_type!r}"
)
if self.capability is not None and self.capability not in KNOWN_CAPABILITIES:
warnings.warn(
Expand Down Expand Up @@ -415,7 +416,7 @@ def get_local_hf_path(self, base_model_name: str) -> str:
base_model_name,
self.repo_id,
revision=self.resolved_revision(),
alora=self.adapter_type is AdapterType.ALORA,
adapter_type=self.adapter_type.value,
)
)

Expand Down Expand Up @@ -770,6 +771,83 @@ def _fire_phase_complete(self, phase: str, duration_s: float) -> None:
)


class PromptBinding(WeightsBinding):
"""Weights binding for the weightless Prompt reality.

A soft adapter with no weights: the capability's `io.yaml` carries a
hand-authored `instruction` that the rewriter injects, and generation runs
against the plain base model. Implemented as a real `WeightsBinding` with no-op
lifecycle verbs (not an optional `Adapter.weights`) so `adapter_scope` and
generation dispatch need no special-casing.

Args:
name (str): Adapter function name (e.g. `"answerability"`).
adapter_type (AdapterType): Always `AdapterType.PROMPT`.
repo_id (str): Hugging Face Hub repository holding the prompt `io.yaml`.
revision (str | None): Git revision, or `None` to resolve the catalogue's
pinned revision lazily.
target_model_name (str): Slot the prompt resolved under — a base model
name, or `"any"`. With `repo_id`/`revision`, used only when a bare

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wording: this says the fetch path is used "when a bare add_adapter(PromptBinding) self-fetches its io.yaml", but AdapterInput is Adapter | _AdapterCore | LocalFileBinding and LocalHFBackend.add_adapter's gate at huggingface.py:2907-2915 accepts only LocalHFAdapter, LocalFileBinding, EmbeddedIntrinsicAdapter or a composed Adapter. A bare PromptBinding is rejected with TypeError.

The reachable shape is a composed Adapter wrapping a PromptBinding, registered without config=. Same wording appears at huggingface.py:1118 and :2937. Worth fixing because these docstrings are published in the API reference.

`add_adapter(PromptBinding)` self-fetches its `io.yaml`; resolution
passes the parsed config instead.

Attributes:
binding_type (ClassVar[str]): `"prompt"`.
"""

binding_type: ClassVar[str] = "prompt"

def __init__(
self,
name: str = "",
adapter_type: AdapterType = AdapterType.PROMPT,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: the Args: entry documents this as "Always AdapterType.PROMPT", but any AdapterType is accepted, and PromptBinding(adapter_type=AdapterType.LORA) would produce qualified_name == "x_lora" on a weightless binding.

The identity agreement check at huggingface.py:2999-3011 catches the mismatched case at registration, so the blast radius is small. A one-line guard or a narrowed annotation would make the docstring true.

repo_id: str = "",
revision: str | None = None,
target_model_name: str = "any",
) -> None:
"""Constructs a PromptBinding."""
self.name = name
self.adapter_type = adapter_type
self.repo_id = repo_id
self.revision = revision
self.target_model_name = target_model_name

@property
def qualified_name(self) -> str:
"""Backend-facing adapter identifier, e.g. `"answerability_prompt"`."""
return f"{self.name}_{self.adapter_type.value}"

def resolved_revision(self) -> str:
"""Returns the revision to download, resolving `None` via the catalogue.

Returns:
The git revision (branch, tag, or commit SHA) to download.

Raises:
ValueError: `revision` is `None` and `name` is not a registered
adapter function.
"""
if self.revision is not None:
return self.revision
return fetch_intrinsic_metadata(self.name).revision

def prepare(self) -> None:
"""No-op: a prompt fallback has no weights to stage."""

def activate(self) -> None:
"""Logs that the soft (prompt) adapter fired, at real-adapter severity."""
MelleaLogger.get_logger().info(
f"Prompt fallback activated for {self.name!r} "
f"(model slot {self.target_model_name!r})."
)

def deactivate(self) -> None:
"""No-op: nothing was loaded, so nothing to unload."""

def release(self) -> None:
"""No-op: a prompt fallback holds no resources to release."""


@dataclass
class EmbeddedActivationRequest:
"""Mutable outgoing-request state that `EmbeddedBinding.apply_activation` edits.
Expand Down Expand Up @@ -866,7 +944,7 @@ async def apply_activation(
invocation-complete event here would have to guess an `outcome` that
this method cannot know, which is worse than not firing it: it would
report `outcome="success"` for calls that go on to fail. Instead, the
caller fires `_fire_embedded_invocation_complete` once generation and
caller fires `_afire_invocation_complete` once generation and
parsing resolve — see its use in `OpenAIBackend`'s and
`LocalHFBackend`'s `granite_formatters_processing` closures (issue
#1560).
Expand Down Expand Up @@ -927,19 +1005,22 @@ async def _fire_activate_phase_complete(self, name: str, duration_s: float) -> N
)


async def _fire_embedded_invocation_complete(
async def _afire_invocation_complete(
*,
identity: Identity,
binding_type: str,
revision: str | None = None,
Comment on lines +1011 to +1012

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a reason to not just pass the WeightsBinding object here instead of its sub-pieces? Similar to how the actual Identity object is getting passed in as well?

outcome: Literal["success", "schema_error", "error"],
error: BaseException | None,
) -> None:
"""Fires `adapter_function_invocation_complete` for an Embedded adapter call.
"""Fires `adapter_function_invocation_complete` from an async generation path.

Called from one of two mutually-exclusive points, since
`EmbeddedBinding.apply_activation` can't know the outcome yet at
request-mutation time (see its docstring): `_await_embedded_generation`
below, on a generation failure, or `granite_formatters_processing` in
`openai.py`/`huggingface.py`, once a response exists.
Used by adapters whose outcome is only known inside a running coroutine —
Embedded and Prompt — where the dispatch/request-mutation site can't know it
yet. Fired from one of two mutually-exclusive points:
`_await_generation_reporting_error` below, on a generation failure, or
`granite_formatters_processing` in `openai.py`/`huggingface.py`, once a
response exists.

Doesn't classify contract-level `IOContract` mismatches on already-valid
JSON — that check runs later, in `call_intrinsic`, after this already
Expand All @@ -951,6 +1032,10 @@ async def _fire_embedded_invocation_complete(

Args:
identity: Identifies the adapter that was invoked.
binding_type: The weights-binding reality that ran (e.g. `"embedded"`,
`"prompt"`).
revision: Git revision the adapter resolved to, or `None` if unpinned or
not applicable.
outcome: The resolved invocation outcome.
error: The exception raised during generation/parsing, or `None` on
success.
Expand All @@ -965,8 +1050,8 @@ async def _fire_embedded_invocation_complete(
try:
payload = AdapterFunctionInvocationCompletePayload(
name=identity.name,
revision=None,
binding_type=EmbeddedBinding.binding_type,
revision=revision,
binding_type=binding_type,
adapter_type=identity.adapter_type,
outcome=outcome,
error=error,
Expand All @@ -981,7 +1066,9 @@ async def _fire_embedded_invocation_complete(
)


async def _await_embedded_generation(coro: Any, identity: Identity) -> Any:
async def _await_generation_reporting_error(
coro: Any, identity: Identity, binding_type: str, revision: str | None = None
) -> Any:
"""Awaits `coro`, firing `outcome="error"` if generation itself fails.

`granite_formatters_processing` only runs once a response object exists,
Expand All @@ -1003,15 +1090,23 @@ async def _await_embedded_generation(coro: Any, identity: Identity) -> Any:
coro: The backend's generation call (e.g. the OpenAI SDK coroutine, or
an `asyncio.to_thread` call wrapping local generation).
identity: Identifies the adapter being invoked.
binding_type: The weights-binding reality that ran (e.g. `"embedded"`,
`"prompt"`).
revision: Git revision the adapter resolved to, or `None` if unpinned or
not applicable.

Returns:
`coro`'s result, unchanged.
"""
try:
return await coro
except BaseException as e:
await _fire_embedded_invocation_complete(
identity=identity, outcome="error", error=e
await _afire_invocation_complete(
identity=identity,
binding_type=binding_type,
revision=revision,
outcome="error",
error=e,
)
raise

Expand Down
Loading
Loading