diff --git a/AGENTS.md b/AGENTS.md index e3e5161cde..90673e341f 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -245,6 +245,7 @@ post-activation shape each produces: | `LocalFileBinding` | LocalFile/PEFT | `prepare` / `activate` / `deactivate` / `release` | `activate()` / `deactivate()`, via `adapter_scope` | Backend-internal PEFT adapter state toggled; the outgoing request is untouched | | `EmbeddedBinding` | Embedded/Granite Switch | none — weights are already in the served model | `apply_activation(request, identity)` | `request.extra_body["chat_template_kwargs"]["adapter_name"]` set; `request.api_params["model"]` removed if present | | `ServerMediatedBinding` | Ollama bundled adapter model | none for the current Ollama path | select the configured model tag during intrinsic generation | Ollama request's `model` is the bundled adapter tag; full lifecycle telemetry remains follow-up work | +| `PromptBinding` | Prompt fallback (weightless) | `prepare` / `activate` / `deactivate` / `release` — all no-ops except `activate()`, which logs | `activate()`, via the backend's weightless generate path (no `adapter_scope`) | None — generation runs against the base model with the rewriter-injected instruction | ### Project Resources diff --git a/docs/docs/advanced/intrinsics.md b/docs/docs/advanced/intrinsics.md index 715ae55fc2..6835eb6973 100644 --- a/docs/docs/advanced/intrinsics.md +++ b/docs/docs/advanced/intrinsics.md @@ -15,6 +15,10 @@ LoRA/aLoRA weights are loaded locally, selected from an embedded checkpoint, or bundled into an Ollama model — faster and more reliable than prompting a general-purpose model for these specialised micro-tasks. +When no trained adapter exists for a capability on the target model, resolution +transparently falls back to a hand-authored prompt run against the base model, so the +capability stays available — see [Prompt fallback](#prompt-fallback). + > **Backend note:** Adapter functions work with three backends: > > - **LocalHFBackend** — loads LoRA/aLoRA adapters from the catalog at runtime. @@ -341,18 +345,40 @@ For a fully custom, non-catalog adapter — your own trained LoRA/aLoRA weights, not one of the built-in adapter functions — see [Adding a custom adapter function in 20 lines](../tutorials/07-custom-adapter-function.md). +## Prompt fallback + +Not every capability has a trained adapter for every model. When no aLoRA or LoRA +adapter exists for a capability on the target model, resolution falls back to a +**prompt**: a hand-authored instruction (packaged as an `io.yaml`) run against the +plain base model. This keeps the capability available — including on models a LoRA +cannot be loaded into — rather than failing. + +Resolution walks the chain in order and uses the first rung that exists: + +1. a model-specific aLoRA adapter, +2. a model-specific LoRA adapter, +3. a model-specific prompt, +4. a model-agnostic `any` prompt (Mellea logs a warning, since an `any` prompt is + less accurate than a model-tuned one). + +The fallback is transparent: the same helper calls (`rag.check_answerability`, +`core.requirement_check`, …) and `Intrinsic` components work unchanged — there is no +adapter to register and nothing to activate. A prompt fallback carries no weights (it +is a `PromptBinding`; see below) and runs the intrinsic's instruction against the base +model, parsing the response through the same output contract a trained adapter uses. + ## Composable adapter construction (advanced) `Adapter` composes an `Identity`, an output contract (`IOContract`), and a weights binding into a single, inspectable object. Both `LocalHFBackend` and `OpenAIBackend` accept a composed `Adapter` directly via `add_adapter` — dispatching on the weights binding's reality (`LocalFileBinding` for -LocalFile/PEFT, `EmbeddedBinding` for Embedded/Granite Switch) — alongside -the deprecated shim classes, which remain functional for now (Epic #929, -issue #1144). An `EmbeddedBinding` adapter additionally requires `add_adapter`'s -`config=` argument (the raw io.yaml mapping) — `add_adapter` raises `ValueError` -without it, since that reality's config cannot be cheaply re-derived later; see -below. +LocalFile/PEFT, `EmbeddedBinding` for Embedded/Granite Switch, `PromptBinding` +for a weightless prompt fallback) — alongside the deprecated shim classes, +which remain functional for now (Epic #929, issue #1144). An `EmbeddedBinding` +adapter additionally requires `add_adapter`'s `config=` argument (the raw +io.yaml mapping) — `add_adapter` raises `ValueError` without it, since that +reality's config cannot be cheaply re-derived later; see below. Each weights binding models how its deployment turns an adapter on. `LocalFileBinding` downloads and loads LoRA/aLoRA weights, so it exposes a @@ -401,13 +427,19 @@ switch_backend.register_embedded_adapter_model( ) ``` +`PromptBinding` has no weights to manage, so there is nothing to load or activate. +Its lifecycle verbs are no-ops (`activate()` only logs that the fallback fired), and +generation runs against the base model with the injected instruction. You never +construct one directly; `resolve_adapter` registers it when the chain lands on a +prompt. + Weights-binding support by backend today: -| Backend | `LocalFileBinding` (LocalFile/PEFT) | `EmbeddedBinding` (Embedded/Granite Switch) | `ServerMediatedBinding` | -| --- | --- | --- | --- | -| `LocalHFBackend` | ✅ shipping — `add_adapter` accepts a composed `Adapter` or a bare `LocalFileBinding` directly | ✅ shipping — `load_embedded_adapters=True`, or `add_adapter(adapter, config=...)`/`register_embedded_adapter_model` with a composed `Adapter` | — | -| `OpenAIBackend` | — | ✅ shipping — `load_embedded_adapters=True`, or `add_adapter(adapter, config=...)`/`register_embedded_adapter_model` with a composed `Adapter` | — | -| `OllamaModelBackend` | — | — | ✅ model selection through `adapter_models` for catalogued adapter functions; lifecycle telemetry is tracked separately | +| Backend | `LocalFileBinding` (LocalFile/PEFT) | `EmbeddedBinding` (Embedded/Granite Switch) | `ServerMediatedBinding` | `PromptBinding` (prompt fallback) | +| --- | --- | --- | --- | --- | +| `LocalHFBackend` | ✅ shipping — `add_adapter` accepts a composed `Adapter` or a bare `LocalFileBinding` directly | ✅ shipping — `load_embedded_adapters=True`, or `add_adapter(adapter, config=...)`/`register_embedded_adapter_model` with a composed `Adapter` | — | ✅ shipping — registered by `resolve_adapter` when the fallback chain lands on a prompt | +| `OpenAIBackend` | — | ✅ shipping — `load_embedded_adapters=True`, or `add_adapter(adapter, config=...)`/`register_embedded_adapter_model` with a composed `Adapter` | — | — | +| `OllamaModelBackend` | — | — | ✅ model selection through `adapter_models` for catalogued adapter functions; lifecycle telemetry is tracked separately | — | `ServerMediatedBinding` currently supports Ollama's bundled-model path. A full server-mediated lifecycle and telemetry contract is tracked separately. diff --git a/docs/docs/advanced/lora-and-alora-adapters.md b/docs/docs/advanced/lora-and-alora-adapters.md index 9165b36daf..6b494f42fa 100644 --- a/docs/docs/advanced/lora-and-alora-adapters.md +++ b/docs/docs/advanced/lora-and-alora-adapters.md @@ -187,11 +187,12 @@ adapter is preferred whenever one is loaded, with three exceptions: 2. The requirement uses the `LLMaJRequirement` subtype explicitly — the caller is asking for LLM-as-a-judge regardless of what adapters are loaded. 3. The adapter is unavailable (e.g. cannot be loaded) — Mellea falls back to - LLM-as-a-judge automatically. This is the *only* fallback case: if the - adapter runs but its output fails schema validation, `validate()` does not - fall back. Instead it surfaces the schema error on `ValidationResult.error` - and fails the check closed (`bool(result)` is `False`), so callers can tell - an unparsable adapter response apart from an ordinary "requirement not met". + LLM-as-a-judge automatically. This is the *only* case where the adapter path + falls back to LLM-as-a-judge: if the adapter runs but its output fails schema + validation, `validate()` does not fall back. Instead it surfaces the schema + error on `ValidationResult.error` and fails the check closed (`bool(result)` + is `False`), so callers can tell an unparsable adapter response apart from an + ordinary "requirement not met". If you want to force the adapter path even when using `generate_from_context` directly (bypassing the normal `validate()` call), use `ALoraRequirement` from diff --git a/mellea/backends/adapters/__init__.py b/mellea/backends/adapters/__init__.py index 1c16445ad1..a3cfbbf312 100644 --- a/mellea/backends/adapters/__init__.py +++ b/mellea/backends/adapters/__init__.py @@ -11,6 +11,7 @@ Identity, IOContract, LocalFileBinding, + PromptBinding, ServerMediatedBinding, WeightsBinding, ) @@ -43,6 +44,7 @@ "IntrinsicAdapter", "LocalFileBinding", "LocalHFAdapter", + "PromptBinding", "ServerMediatedBinding", "WeightsBinding", "fetch_intrinsic_metadata", diff --git a/mellea/backends/adapters/_core.py b/mellea/backends/adapters/_core.py index 40e45ae3cf..27d2d76e3e 100644 --- a/mellea/backends/adapters/_core.py +++ b/mellea/backends/adapters/_core.py @@ -104,22 +104,23 @@ class Identity: Attributes: name (str): Human-readable adapter name. - adapter_type (Literal["lora", "alora"]): The LoRA variant. + adapter_type (Literal["lora", "alora", "prompt"]): The adapter variant. `"prompt"` is the + weightless prompt fallback. capability (str | None): Advisory capability string; emits :class:`UserWarning` when not in :data:`~mellea.backends.adapters.capabilities.KNOWN_CAPABILITIES`. """ name: str - adapter_type: Literal["lora", "alora"] + adapter_type: Literal["lora", "alora", "prompt"] capability: str | None = None def __post_init__(self) -> None: # Literal[...] is a static-only constraint; mypy enforces it but Python # does not, so validate at runtime too. - if self.adapter_type not in ("lora", "alora"): + if self.adapter_type not in ("lora", "alora", "prompt"): raise ValueError( - f"adapter_type must be 'lora' or 'alora', got {self.adapter_type!r}" + f"adapter_type must be 'lora', 'alora', or 'prompt', got {self.adapter_type!r}" ) if self.capability is not None and self.capability not in KNOWN_CAPABILITIES: warnings.warn( @@ -415,7 +416,7 @@ def get_local_hf_path(self, base_model_name: str) -> str: base_model_name, self.repo_id, revision=self.resolved_revision(), - alora=self.adapter_type is AdapterType.ALORA, + adapter_type=self.adapter_type.value, ) ) @@ -770,6 +771,83 @@ def _fire_phase_complete(self, phase: str, duration_s: float) -> None: ) +class PromptBinding(WeightsBinding): + """Weights binding for the weightless Prompt reality. + + A soft adapter with no weights: the capability's `io.yaml` carries a + hand-authored `instruction` that the rewriter injects, and generation runs + against the plain base model. Implemented as a real `WeightsBinding` with no-op + lifecycle verbs (not an optional `Adapter.weights`) so `adapter_scope` and + generation dispatch need no special-casing. + + Args: + name (str): Adapter function name (e.g. `"answerability"`). + adapter_type (AdapterType): Always `AdapterType.PROMPT`. + repo_id (str): Hugging Face Hub repository holding the prompt `io.yaml`. + revision (str | None): Git revision, or `None` to resolve the catalogue's + pinned revision lazily. + target_model_name (str): Slot the prompt resolved under — a base model + name, or `"any"`. With `repo_id`/`revision`, used only when a bare + `add_adapter(PromptBinding)` self-fetches its `io.yaml`; resolution + passes the parsed config instead. + + Attributes: + binding_type (ClassVar[str]): `"prompt"`. + """ + + binding_type: ClassVar[str] = "prompt" + + def __init__( + self, + name: str = "", + adapter_type: AdapterType = AdapterType.PROMPT, + repo_id: str = "", + revision: str | None = None, + target_model_name: str = "any", + ) -> None: + """Constructs a PromptBinding.""" + self.name = name + self.adapter_type = adapter_type + self.repo_id = repo_id + self.revision = revision + self.target_model_name = target_model_name + + @property + def qualified_name(self) -> str: + """Backend-facing adapter identifier, e.g. `"answerability_prompt"`.""" + return f"{self.name}_{self.adapter_type.value}" + + def resolved_revision(self) -> str: + """Returns the revision to download, resolving `None` via the catalogue. + + Returns: + The git revision (branch, tag, or commit SHA) to download. + + Raises: + ValueError: `revision` is `None` and `name` is not a registered + adapter function. + """ + if self.revision is not None: + return self.revision + return fetch_intrinsic_metadata(self.name).revision + + def prepare(self) -> None: + """No-op: a prompt fallback has no weights to stage.""" + + def activate(self) -> None: + """Logs that the soft (prompt) adapter fired, at real-adapter severity.""" + MelleaLogger.get_logger().info( + f"Prompt fallback activated for {self.name!r} " + f"(model slot {self.target_model_name!r})." + ) + + def deactivate(self) -> None: + """No-op: nothing was loaded, so nothing to unload.""" + + def release(self) -> None: + """No-op: a prompt fallback holds no resources to release.""" + + @dataclass class EmbeddedActivationRequest: """Mutable outgoing-request state that `EmbeddedBinding.apply_activation` edits. @@ -866,7 +944,7 @@ async def apply_activation( invocation-complete event here would have to guess an `outcome` that this method cannot know, which is worse than not firing it: it would report `outcome="success"` for calls that go on to fail. Instead, the - caller fires `_fire_embedded_invocation_complete` once generation and + caller fires `_afire_invocation_complete` once generation and parsing resolve — see its use in `OpenAIBackend`'s and `LocalHFBackend`'s `granite_formatters_processing` closures (issue #1560). @@ -927,19 +1005,22 @@ async def _fire_activate_phase_complete(self, name: str, duration_s: float) -> N ) -async def _fire_embedded_invocation_complete( +async def _afire_invocation_complete( *, identity: Identity, + binding_type: str, + revision: str | None = None, outcome: Literal["success", "schema_error", "error"], error: BaseException | None, ) -> None: - """Fires `adapter_function_invocation_complete` for an Embedded adapter call. + """Fires `adapter_function_invocation_complete` from an async generation path. - Called from one of two mutually-exclusive points, since - `EmbeddedBinding.apply_activation` can't know the outcome yet at - request-mutation time (see its docstring): `_await_embedded_generation` - below, on a generation failure, or `granite_formatters_processing` in - `openai.py`/`huggingface.py`, once a response exists. + Used by adapters whose outcome is only known inside a running coroutine — + Embedded and Prompt — where the dispatch/request-mutation site can't know it + yet. Fired from one of two mutually-exclusive points: + `_await_generation_reporting_error` below, on a generation failure, or + `granite_formatters_processing` in `openai.py`/`huggingface.py`, once a + response exists. Doesn't classify contract-level `IOContract` mismatches on already-valid JSON — that check runs later, in `call_intrinsic`, after this already @@ -951,6 +1032,10 @@ async def _fire_embedded_invocation_complete( Args: identity: Identifies the adapter that was invoked. + binding_type: The weights-binding reality that ran (e.g. `"embedded"`, + `"prompt"`). + revision: Git revision the adapter resolved to, or `None` if unpinned or + not applicable. outcome: The resolved invocation outcome. error: The exception raised during generation/parsing, or `None` on success. @@ -965,8 +1050,8 @@ async def _fire_embedded_invocation_complete( try: payload = AdapterFunctionInvocationCompletePayload( name=identity.name, - revision=None, - binding_type=EmbeddedBinding.binding_type, + revision=revision, + binding_type=binding_type, adapter_type=identity.adapter_type, outcome=outcome, error=error, @@ -981,7 +1066,9 @@ async def _fire_embedded_invocation_complete( ) -async def _await_embedded_generation(coro: Any, identity: Identity) -> Any: +async def _await_generation_reporting_error( + coro: Any, identity: Identity, binding_type: str, revision: str | None = None +) -> Any: """Awaits `coro`, firing `outcome="error"` if generation itself fails. `granite_formatters_processing` only runs once a response object exists, @@ -1003,6 +1090,10 @@ async def _await_embedded_generation(coro: Any, identity: Identity) -> Any: coro: The backend's generation call (e.g. the OpenAI SDK coroutine, or an `asyncio.to_thread` call wrapping local generation). identity: Identifies the adapter being invoked. + binding_type: The weights-binding reality that ran (e.g. `"embedded"`, + `"prompt"`). + revision: Git revision the adapter resolved to, or `None` if unpinned or + not applicable. Returns: `coro`'s result, unchanged. @@ -1010,8 +1101,12 @@ async def _await_embedded_generation(coro: Any, identity: Identity) -> Any: try: return await coro except BaseException as e: - await _fire_embedded_invocation_complete( - identity=identity, outcome="error", error=e + await _afire_invocation_complete( + identity=identity, + binding_type=binding_type, + revision=revision, + outcome="error", + error=e, ) raise diff --git a/mellea/backends/adapters/adapter.py b/mellea/backends/adapters/adapter.py index c4a53f0573..e78edd3a6c 100644 --- a/mellea/backends/adapters/adapter.py +++ b/mellea/backends/adapters/adapter.py @@ -30,6 +30,7 @@ from ...core import Backend, MelleaLogger from ...formatters.granite import intrinsics as intrinsics +from ...formatters.granite.intrinsics import AdapterNotFoundError from ...helpers.event_loop_helper import _run_async_in_thread from ...plugins.manager import has_plugins, invoke_hook from ...plugins.types import HookType @@ -39,9 +40,15 @@ EmbeddedBinding, Identity, LocalFileBinding, + PromptBinding, WeightsBinding, ) -from .catalog import AdapterType, fetch_intrinsic_metadata, known_intrinsic_names +from .catalog import ( + AdapterType, + IntrinsicsCatalogEntry, + fetch_intrinsic_metadata, + known_intrinsic_names, +) from .io_contracts import get_io_contract @@ -219,17 +226,15 @@ def __init__( assert self.base_model_name is not None, ( "must provide `base_model_name` if not providing a `config_file` or `config_dict`" ) - # We're converting the adapter type to a boolean flag here. assert adapter_type in (AdapterType.ALORA, AdapterType.LORA), ( f"{adapter_type} not supported" ) - is_alora = self.adapter_type == AdapterType.ALORA config_file = intrinsics.obtain_io_yaml( self.intrinsic_name, self.base_model_name, self.intrinsic_metadata.repo_id, revision=self.intrinsic_metadata.revision, - alora=is_alora, + adapter_type=self.adapter_type.value, ) if config_file: with open(config_file, encoding="utf-8") as f: @@ -283,14 +288,13 @@ def download_and_get_path(self, base_model_name: str) -> str: Returns: a path to the files """ - is_alora = self.adapter_type == AdapterType.ALORA return str( intrinsics.obtain_lora( self.intrinsic_name, base_model_name, self.intrinsic_metadata.repo_id, revision=self.intrinsic_metadata.revision, - alora=is_alora, + adapter_type=self.adapter_type.value, ) ) @@ -513,6 +517,99 @@ def _fire_invocation_complete( AdapterInput: TypeAlias = Adapter | _AdapterCore | LocalFileBinding +def _register_available_composed_adapter( + backend: "AdapterMixin", name: str, base: str, metadata: IntrinsicsCatalogEntry +) -> None: + """Register the first adapter in the fallback chain that exists for `base`. + + Probes aLoRA, LoRA, a model-specific prompt, then the `"any"` prompt via a tiny + `io.yaml` fetch — advancing only on the not-found `AdapterNotFoundError`, letting + other errors propagate — and registers a `LocalFileBinding` (trained) or a + `PromptBinding` (prompt, warning on `"any"`) for the first hit. Trial-fetch is + the only per-model presence signal; the catalogue's `adapter_types` is + per-capability. + + Module-level, not a method, so a spec'd mock backend in tests drives the real + walk. + + Raises: + AdapterNotFoundError: No adapter or prompt fallback exists for `name` on + `base`. + """ + for adapter_type in (AdapterType.ALORA, AdapterType.LORA): + try: + intrinsics.obtain_io_yaml( + name, + base, + metadata.repo_id, + revision=metadata.revision, + adapter_type=adapter_type.value, + ) + except AdapterNotFoundError: + continue + backend.add_adapter( + _AdapterCore( + identity=Identity( + name=name, + adapter_type=adapter_type.value, # type: ignore[arg-type] + capability=metadata.effective_capability, + ), + io_contract=get_io_contract(name), + weights=LocalFileBinding( + name=name, + adapter_type=adapter_type, + repo_id=metadata.repo_id, + revision=metadata.revision, + ), + ) + ) + return + + try: + io_yaml_path, used_any = intrinsics.resolve_prompt_io_yaml( + name, base, metadata.repo_id, revision=metadata.revision + ) + except AdapterNotFoundError as e: + raise AdapterNotFoundError( + f"No adapter (aLoRA/LoRA) or prompt fallback found for adapter " + f"function {name!r} on base model {base!r} in {metadata.repo_id}." + ) from e + + target_model = "any" if used_any else base + if used_any: + MelleaLogger.get_logger().warning( + f"No adapter or model-specific prompt found for adapter function " + f"{name!r} on base model {base!r}; falling back to the " + "model-agnostic 'any' prompt, which is sub-par relative to a " + "model-specific one." + ) + with open(io_yaml_path, encoding="utf-8") as f: + prompt_config = yaml.safe_load(f) + if not isinstance(prompt_config, dict): + raise ValueError( + f"Prompt io.yaml for {name!r} at {io_yaml_path} did not parse " + f"to a mapping (got {type(prompt_config).__name__})." + ) + backend.add_adapter( + _AdapterCore( + identity=Identity( + name=name, + adapter_type="prompt", + capability=metadata.effective_capability, + ), + io_contract=get_io_contract(name), + weights=PromptBinding( + name=name, + adapter_type=AdapterType.PROMPT, + repo_id=metadata.repo_id, + revision=metadata.revision, + target_model_name=target_model, + ), + ), + config=prompt_config, + ) + + class AdapterMixin(Backend, abc.ABC): """Mixin class for backends capable of utilizing adapters. @@ -818,7 +915,9 @@ def resolve_adapter(self, name: str) -> _AdapterCore: _AdapterCore: The registered adapter with the given capability. Raises: - ValueError: If the backend has no model ID. + ValueError: If the backend has no model ID, or (for a backend that + supports composed adapters) if no trained adapter and no prompt + fallback exists for `name` on the backend's base model. KeyError: If the adapter cannot be found after registration. Note: @@ -893,29 +992,11 @@ def resolve_adapter(self, name: str) -> _AdapterCore: ): self._add_embedded_adapter_compat(a, config) else: - # AdapterType.LORA is the pre-Phase-1 default (mirrors old _util.py). - # Every current catalog entry supports LORA. Phase 2 (see epic #929) - # will select the type from catalog availability instead of hardcoding. metadata = fetch_intrinsic_metadata(name) if self._supports_composed_adapters: - # Composed Adapter, not the deprecated IntrinsicAdapter shim - # (Epic #929, issue #1144). - self.add_adapter( - _AdapterCore( - identity=Identity( - name=name, - adapter_type="lora", - capability=metadata.effective_capability, - ), - io_contract=get_io_contract(name), - weights=LocalFileBinding( - name=name, - adapter_type=AdapterType.LORA, - repo_id=metadata.repo_id, - revision=metadata.revision, - ), - ) - ) + # Register whichever adapter (or prompt) actually exists for + # this base model. + _register_available_composed_adapter(self, name, base, metadata) else: # Legacy subclass predating the composed-Adapter contract # (same reasoning as _add_embedded_adapter_compat): its diff --git a/mellea/backends/adapters/catalog.py b/mellea/backends/adapters/catalog.py index 426d06d4c1..3b236785df 100644 --- a/mellea/backends/adapters/catalog.py +++ b/mellea/backends/adapters/catalog.py @@ -45,10 +45,12 @@ class AdapterType(enum.Enum): Attributes: LORA (str): Standard LoRA adapter; value `"lora"`. ALORA (str): aLoRA adapter (shares model KV cache across adapter functions); value `"alora"`. + PROMPT (str): Weightless prompt fallback used when no trained adapter exists; value `"prompt"`. """ LORA = "lora" ALORA = "alora" + PROMPT = "prompt" class IntrinsicsCatalogEntry(pydantic.BaseModel): diff --git a/mellea/backends/huggingface.py b/mellea/backends/huggingface.py index 910c166f1d..3208d31577 100644 --- a/mellea/backends/huggingface.py +++ b/mellea/backends/huggingface.py @@ -93,7 +93,6 @@ AdapterMixin, EmbeddedActivationRequest, EmbeddedBinding, - Identity, IntrinsicAdapter, LocalHFAdapter, ) @@ -101,9 +100,10 @@ Adapter as _AdapterCore, IOContract, LocalFileBinding, + PromptBinding, WeightsBinding, - _await_embedded_generation, - _fire_embedded_invocation_complete, + _afire_invocation_complete, + _await_generation_reporting_error, ) from .adapters.adapter import ( AdapterInput, @@ -111,7 +111,6 @@ _composed_adapter_key, _discover_embedded_adapters, ) -from .adapters.catalog import AdapterType from .backend import FormatterBackend from .cache import Cache, SimpleLRUCache from .model_ids import ModelIdentifier @@ -977,6 +976,49 @@ def _generate_composed_local_file_with_adapter_scope( _assert_correct_adapters(binding.qualified_name, self._model) return out + def _generate_composed_prompt( + self, + adapter: _AdapterCore, + generate_func: Callable[..., _T], + *args: Any, + **kwargs: Any, + ) -> _T: + """Runs `generate_func` for a composed `Adapter`'s weightless `PromptBinding`. + + A prompt fallback has no weights: `IntrinsicsRewriter` already injected the + instruction, so generation runs against the plain base model with no + `adapter_scope`. It fires the binding's `activate()` (which just logs), then + serialises on `_generation_lock` and ensures no PEFT adapter is active first. + + Args: + adapter: The composed `Adapter` to run; its `weights` must be a + `PromptBinding`. + generate_func: The synchronous generation callable to invoke. + *args: Positional arguments forwarded to `generate_func`. + **kwargs: Keyword arguments forwarded to `generate_func`. + + Returns: + Whatever `generate_func` returns. + + Raises: + TypeError: `adapter.weights` is not a `PromptBinding`. + """ + binding = adapter.weights + if not isinstance(binding, PromptBinding): + raise TypeError( + "LocalHFBackend's composed prompt generation path requires " + f"a PromptBinding; got {type(binding).__name__}." + ) + with self._generation_lock: + # No PEFT adapter is active on this path; deactivate defensively and + # assert the base model is clean, matching the embedded path. + self.deactivate_peft_adapter("") + _assert_correct_adapters("", self._model) + binding.activate() + out = generate_func(*args, **kwargs) + _assert_correct_adapters("", self._model) + return out + def _intrinsic_adapter_name_and_config( self, adapter: IntrinsicAdapter | EmbeddedIntrinsicAdapter | _AdapterCore ) -> tuple[str, dict]: @@ -991,8 +1033,10 @@ def _intrinsic_adapter_name_and_config( branch (via `_obtain_local_file_io_yaml_config`), alongside the weights download `binding.prepare()` already does there; the Embedded/Granite Switch reality is cached at registration time (see - `add_adapter`/`register_embedded_adapter_model`). Deliberately not - fetched here: this method runs inline inside the async + `add_adapter`/`register_embedded_adapter_model`); the weightless Prompt + reality is cached by `add_adapter`'s prompt branch (config supplied by + `resolve_adapter`, or fetched via `_obtain_prompt_io_yaml_config`). + Deliberately not fetched here: this method runs inline inside the async `_generate_from_intrinsic`, and `obtain_io_yaml`'s Hugging Face Hub round trip must not block the event loop. @@ -1004,15 +1048,17 @@ def _intrinsic_adapter_name_and_config( io.yaml config. Raises: - ValueError: A composed LocalFile or Embedded adapter has no + ValueError: A composed LocalFile, Embedded, or Prompt adapter has no cached config (never registered via `add_adapter`/`register_embedded_adapter_model`). - TypeError: A composed Adapter's `weights` is neither a - `LocalFileBinding` nor an `EmbeddedBinding`. + TypeError: A composed Adapter's `weights` is not a + `LocalFileBinding`, `EmbeddedBinding`, or `PromptBinding`. """ if isinstance(adapter, (IntrinsicAdapter, EmbeddedIntrinsicAdapter)): return adapter.name, adapter.config - if isinstance(adapter.weights, (LocalFileBinding, EmbeddedBinding)): + if isinstance( + adapter.weights, (LocalFileBinding, EmbeddedBinding, PromptBinding) + ): key = _composed_adapter_key(adapter) config = self._composed_adapter_configs.get(key) if config is None: @@ -1052,7 +1098,7 @@ def _obtain_local_file_io_yaml_config(self, binding: LocalFileBinding) -> dict: self.base_model_name, binding.repo_id, revision=binding.resolved_revision(), - alora=binding.adapter_type is AdapterType.ALORA, + adapter_type=binding.adapter_type.value, ) with open(io_yaml_path, encoding="utf-8") as f: loaded = yaml.safe_load(f) @@ -1063,6 +1109,41 @@ def _obtain_local_file_io_yaml_config(self, binding: LocalFileBinding) -> dict: ) return loaded + def _obtain_prompt_io_yaml_config(self, binding: PromptBinding) -> dict: + """Download and parse a prompt `binding`'s io.yaml. + + The weightless counterpart to `_obtain_local_file_io_yaml_config`: a prompt + binding has no weights, so this fetch is the only I/O its registration does. + It reads from the binding's resolved model slot (`binding.target_model_name`, + a base model name or the literal `"any"`) under the `prompt` subdir. Only + used for a bare `add_adapter(PromptBinding)` call; `resolve_adapter` passes + the already-parsed config via `config=` and this is never reached. + + Args: + binding: The `PromptBinding` to load `io.yaml` for. + + Returns: + dict: The parsed `io.yaml` mapping. + + Raises: + ValueError: `io.yaml` did not parse to a mapping. + """ + io_yaml_path = intrinsics.obtain_io_yaml( + binding.name, + binding.target_model_name, + binding.repo_id, + revision=binding.resolved_revision(), + adapter_type="prompt", + ) + with open(io_yaml_path, encoding="utf-8") as f: + loaded = yaml.safe_load(f) + if not isinstance(loaded, dict): + raise ValueError( + f"io.yaml for prompt adapter {binding.name!r} at {io_yaml_path} " + f"did not parse to a mapping (got {type(loaded).__name__})." + ) + return loaded + async def _generate_from_intrinsic( self, action: Intrinsic, @@ -1157,8 +1238,10 @@ async def _generate_from_intrinsic( docs = messages_to_docs(ctx_as_message_list) - allowed_types = tuple(at.value for at in action.adapter_types) - adapter = self._find_adapter(action.intrinsic_name, allowed_types) + # resolve_adapter registers exactly one winner per capability, so match by + # name only: pass None (not action.adapter_types, which is never None) so a + # weightless PromptBinding winner isn't filtered out by adapter type. + adapter = self._find_adapter(action.intrinsic_name, None) if adapter is None: raise ValueError( f"backend ({self}) has no adapter for processing intrinsic: {action.intrinsic_name}" @@ -1234,10 +1317,6 @@ async def _generate_from_intrinsic( ), adapter.identity, ) - # Reused below (generation-path dispatch, embedded_identity for - # post-processing) — computed once here rather than re-checked, since - # unlike the `if` above this isn't gating a narrowing-sensitive block. - adapter_is_embedded = isinstance(adapter_weights, EmbeddedBinding) generate_input, other_input = ( granite_formatters.base.util.chat_completion_request_to_transformers_inputs( # type: ignore @@ -1301,8 +1380,8 @@ def __getattr__(self_proxy, name: str) -> Any: model_arg = _CapturingModelProxy() # type: ignore[assignment] - if adapter_is_embedded: - chat_response = _await_embedded_generation( + if isinstance(adapter.weights, EmbeddedBinding): + chat_response = _await_generation_reporting_error( asyncio.to_thread( self._generate_embedded_with_generation_lock, granite_formatters.base.util.generate_with_transformers, # type: ignore @@ -1313,6 +1392,7 @@ def __getattr__(self_proxy, name: str) -> Any: other_input, ), adapter.identity, + adapter.weights.binding_type, ) elif isinstance(adapter, IntrinsicAdapter): chat_response = asyncio.to_thread( @@ -1325,6 +1405,22 @@ def __getattr__(self_proxy, name: str) -> Any: generate_input, other_input, ) + elif isinstance(adapter.weights, PromptBinding): + chat_response = _await_generation_reporting_error( + asyncio.to_thread( + self._generate_composed_prompt, + adapter, + granite_formatters.base.util.generate_with_transformers, # type: ignore + # Passed as args/kwargs to generate. + self._tokenizer, + model_arg, + generate_input, + other_input, + ), + adapter.identity, + adapter.weights.binding_type, + adapter.weights.revision, + ) else: chat_response = asyncio.to_thread( self._generate_composed_local_file_with_adapter_scope, @@ -1350,13 +1446,10 @@ async def granite_formatters_processing( rewritten: granite_formatters.ChatCompletion, result_processor: granite_formatters.IntrinsicsResultProcessor, input_ids, - embedded_identity: Identity | None, ): - # embedded_identity is set only for EmbeddedIntrinsicAdapter calls - # (the legacy PEFT path already gets this signal via adapter_scope). - # Kept in one try so any failure below — not just from transform - # itself — fires outcome="error" instead of reporting success and - # then raising, the exact bug #1559 reverted. + # One try/finally: a failure anywhere in processing (not just in + # transform) must be reported as an error, never success-then-raise. + error: BaseException | None = None try: res = result_processor.transform(chunk, rewritten) # type: ignore @@ -1378,34 +1471,40 @@ async def granite_formatters_processing( ) # processing expects a str or a GenerateDecoderOnlyOutput. Extract the str. - result = await self.processing( + return await self.processing( mot, res.choices[0].message.content, input_ids=input_ids ) except json.JSONDecodeError as e: - if embedded_identity is not None: - await _fire_embedded_invocation_complete( - identity=embedded_identity, outcome="schema_error", error=e - ) + error = e raise Exception(f"Intrinsic did not return a JSON: {chunk}") from e - except Exception as e: - if embedded_identity is not None: - await _fire_embedded_invocation_complete( - identity=embedded_identity, outcome="error", error=e - ) + except BaseException as e: + error = e raise - else: - if embedded_identity is not None: - await _fire_embedded_invocation_complete( - identity=embedded_identity, outcome="success", error=None + finally: + if isinstance(adapter.weights, (EmbeddedBinding, PromptBinding)): + await _afire_invocation_complete( + identity=adapter.identity, + binding_type=adapter.weights.binding_type, + revision=( + adapter.weights.revision + if isinstance(adapter.weights, PromptBinding) + else None + ), + outcome=( + "success" + if error is None + else "schema_error" + if isinstance(error, json.JSONDecodeError) + else "error" + ), + error=error, ) - return result output._gen.process = functools.partial( granite_formatters_processing, rewritten=rewritten, result_processor=result_processor, input_ids=generate_input["input_tokens"], - embedded_identity=(adapter.identity if adapter_is_embedded else None), ) # TODO: Post-processing should release the lock for this generation. @@ -2834,10 +2933,14 @@ def add_adapter(self, adapter: AdapterInput, *, config: dict | None = None) -> N "Pass config=, or register it via " "register_embedded_adapter_model() or resolve_adapter()." ) + elif isinstance(adapter.weights, PromptBinding): + # config= is optional: resolve_adapter passes the fetched io.yaml; + # a bare add_adapter(PromptBinding) self-fetches it below. + pass elif config is not None: raise TypeError( "config= is only accepted for a composed Adapter whose " - f"weights is an EmbeddedBinding; got " + f"weights is an EmbeddedBinding or PromptBinding; got " f"{type(adapter.weights).__name__}, which derives its " "config lazily from io.yaml on first use." ) @@ -2886,7 +2989,9 @@ def add_adapter(self, adapter: AdapterInput, *, config: dict | None = None) -> N self._composed_adapters[key] = adapter self._composed_adapter_configs[key] = config return # atomic: no lifecycle verb, no I/O - if not isinstance(adapter.weights, LocalFileBinding): + if not isinstance( + adapter.weights, (LocalFileBinding, PromptBinding) + ): raise TypeError( "LocalHFBackend does not support the " f"{type(adapter.weights).__name__} weights reality " @@ -2906,6 +3011,20 @@ def add_adapter(self, adapter: AdapterInput, *, config: dict | None = None) -> N "different strings for the same adapter (NOTE(#1516))." ) + # A PromptBinding is weightless — no bind_backend()/prepare(): just + # register it and cache its io.yaml (from resolve_adapter, or + # self-fetched here). Any fetch stays outside the activation lock. + if isinstance(binding, PromptBinding): + prompt_config = ( + config + if config is not None + else self._obtain_prompt_io_yaml_config(binding) + ) + with self._adapter_activation_lock(): + self._composed_adapter_configs[key] = prompt_config + self._composed_adapters[key] = adapter + return + # Slow work, still under the resolve lock but not the # activation lock: released above, so bind_backend()/ # prepare()'s own lifecycle-then-activation acquisitions diff --git a/mellea/backends/ollama.py b/mellea/backends/ollama.py index d03077c4fc..0ec54c2e8c 100644 --- a/mellea/backends/ollama.py +++ b/mellea/backends/ollama.py @@ -461,7 +461,7 @@ def _resolve_adapter_uncoalesced(self, name: str) -> _AdapterCore: self.base_model_name, metadata.repo_id, revision=metadata.revision, - alora=use_alora, + adapter_type=adapter_type.value, ) except ModuleNotFoundError as e: if e.name != "huggingface_hub": diff --git a/mellea/backends/openai.py b/mellea/backends/openai.py index 41f2d3a0e7..66e9aa4247 100644 --- a/mellea/backends/openai.py +++ b/mellea/backends/openai.py @@ -59,8 +59,8 @@ from .adapters import EmbeddedActivationRequest, EmbeddedBinding, Identity from .adapters._core import ( Adapter as _AdapterCore, - _await_embedded_generation, - _fire_embedded_invocation_complete, + _afire_invocation_complete, + _await_generation_reporting_error, ) from .adapters.adapter import ( AdapterInput, @@ -1210,7 +1210,7 @@ async def _generate_from_intrinsic( d["role"] = m.role messages_dicts.append(d) - chat_response = _await_embedded_generation( + chat_response = _await_generation_reporting_error( self._async_client.chat.completions.create( model=self._model_id, messages=messages_dicts, # type: ignore @@ -1219,6 +1219,7 @@ async def _generate_from_intrinsic( **api_params, ), adapter.identity, + adapter.weights.binding_type, ) # --- wire up ModelOutputThunk with intrinsic post-processing ------ @@ -1234,6 +1235,7 @@ async def granite_formatters_processing( rewritten: granite_formatters.ChatCompletion, result_processor: granite_formatters.IntrinsicsResultProcessor, identity: Identity, + binding_type: str, ): """Accumulate content and apply intrinsic result processing.""" import json as _json @@ -1242,7 +1244,8 @@ async def granite_formatters_processing( # response with an empty choices list raises IndexError there # (openai.py's processing() indexes chunk.choices[0].message # unguarded), and that must still fire outcome="error" rather - # than skip the fire site entirely. + # than skip the emit entirely. + error: BaseException | None = None try: # Delegate standard metadata storage to the shared processing method. await self.processing(mot, chunk) @@ -1254,21 +1257,26 @@ async def granite_formatters_processing( # then raise — the exact bug #1559 reverted. mot._underlying_value = res.choices[0].message.content except _json.JSONDecodeError as e: - await _fire_embedded_invocation_complete( - identity=identity, outcome="schema_error", error=e - ) + error = e raise Exception( f"Intrinsic did not return a JSON: " f"{chunk.choices[0].message.content}" ) from e - except Exception as e: - await _fire_embedded_invocation_complete( - identity=identity, outcome="error", error=e - ) + except BaseException as e: + error = e raise - else: - await _fire_embedded_invocation_complete( - identity=identity, outcome="success", error=None + finally: + await _afire_invocation_complete( + identity=identity, + binding_type=binding_type, + outcome=( + "success" + if error is None + else "schema_error" + if isinstance(error, _json.JSONDecodeError) + else "error" + ), + error=error, ) # Processing functions only pass the ModelOutputThunk (and current chunk @@ -1278,6 +1286,7 @@ async def granite_formatters_processing( rewritten=rewritten, result_processor=result_processor, identity=adapter.identity, + binding_type=adapter.weights.binding_type, ) output._gen.post_process = functools.partial( diff --git a/mellea/formatters/granite/intrinsics/__init__.py b/mellea/formatters/granite/intrinsics/__init__.py index 71a7982250..f25bc10a33 100644 --- a/mellea/formatters/granite/intrinsics/__init__.py +++ b/mellea/formatters/granite/intrinsics/__init__.py @@ -6,11 +6,18 @@ # Local from .input import IntrinsicsRewriter from .output import IntrinsicsResultProcessor -from .util import obtain_io_yaml, obtain_lora +from .util import ( + AdapterNotFoundError, + obtain_io_yaml, + obtain_lora, + resolve_prompt_io_yaml, +) __all__ = ( + "AdapterNotFoundError", "IntrinsicsResultProcessor", "IntrinsicsRewriter", "obtain_io_yaml", "obtain_lora", + "resolve_prompt_io_yaml", ) diff --git a/mellea/formatters/granite/intrinsics/util.py b/mellea/formatters/granite/intrinsics/util.py index 7743a80573..4d007cdcb4 100644 --- a/mellea/formatters/granite/intrinsics/util.py +++ b/mellea/formatters/granite/intrinsics/util.py @@ -22,6 +22,15 @@ ) +class AdapterNotFoundError(ValueError): + """Raised when no adapter or prompt `io.yaml` exists at the expected repository path. + + Subclasses `ValueError` so a broad `except ValueError` still catches it, while + callers can catch `AdapterNotFoundError` specifically to treat "not found" as a + fallback signal without also swallowing other `ValueError`s. + """ + + def make_config_dict( config_file: str | pathlib.Path | None = None, config_dict: dict | None = None ) -> dict | None: @@ -99,38 +108,42 @@ def make_config_dict( def adapter_subpath( - intrinsic_name: str, target_model_name: str, repo_id: str, /, alora: bool = False + intrinsic_name: str, + target_model_name: str, + repo_id: str, + /, + adapter_type: str = "lora", ) -> str: """Return the Hugging Face Hub subpath where an intrinsic's adapter lives. Encapsulates the layout convention used by the Granite Intrinsics Library and - related repositories so callers don't replicate the rules. Both - :func:`obtain_lora` and out-of-tree consumers (e.g. drift checks in tests) - should call this function rather than building the path themselves. + related repositories so callers don't replicate the rules. Both `obtain_lora` + and out-of-tree consumers (e.g. drift checks in tests) should call this function + rather than building the path themselves. Args: intrinsic_name: Short name of the intrinsic model, such as `"certainty"`. - target_model_name: Name of the base model for the LoRA or aLoRA adapter. - May be a raw HF repo ID; canonical normalization is applied. + target_model_name: Name of the base model for the adapter, or `"any"` for a + model-agnostic prompt fallback. May be a raw HF repo ID; canonical + normalization is applied (`"any"` is not normalized and passes through). repo_id: Hugging Face Hub repository containing the adapter collection. Used to select between old and new directory layouts. - alora: If `True`, return the path for the aLoRA variant; otherwise LoRA. + adapter_type: Adapter-type slot — `"lora"`, `"alora"`, or `"prompt"`. Returns: - Subpath relative to the repo root, e.g. `"certainty/granite-4.1-3b/lora"`. + Subpath relative to the repo root, e.g. `"certainty/granite-4.1-3b/lora"` or + `"answerability/any/prompt"`. """ # Normalize target model name if a normalization exists. target_model_name = BASE_MODEL_TO_CANONICAL_NAME.get( target_model_name, target_model_name ) - lora_str = "alora" if alora else "lora" - if repo_id in OLD_LAYOUT_REPOS: # Old repository layout. - return f"{intrinsic_name}/{lora_str}/{target_model_name}" + return f"{intrinsic_name}/{adapter_type}/{target_model_name}" - return f"{intrinsic_name}/{target_model_name}/{lora_str}" + return f"{intrinsic_name}/{target_model_name}/{adapter_type}" def obtain_lora( @@ -139,44 +152,45 @@ def obtain_lora( repo_id: str, /, revision: str = "main", - alora: bool = False, cache_dir: str | None = None, file_glob: str = "*", + adapter_type: str = "lora", ) -> pathlib.Path: - """Download and cache an adapter that implements and intrinsic. + """Download and cache an adapter that implements an intrinsic. - Downloads a LoRA or aLoRA adapter from a collection of adapters that follow the - same layout as the [Granite Intrinsics Library]( + Downloads a LoRA, aLoRA, or prompt adapter from a collection of adapters that + follow the same layout as the [Granite Intrinsics Library]( https://huggingface.co/ibm-granite/granitelib-rag-r1.0). Caches the downloaded adapter files on local disk. Args: intrinsic_name: Short name of the intrinsic model, such as `"certainty"`. - target_model_name: Name of the base model for the LoRA or aLoRA adapter. + target_model_name: Name of the base model for the adapter, or `"any"` for a + model-agnostic prompt fallback. repo_id: Hugging Face Hub repository containing a collection of LoRA and/or aLoRA adapters for intrinsics. revision: Git revision of the repository to download from. - alora: If `True`, load the aLoRA version of the intrinsic; otherwise use LoRA. cache_dir: Local directory to use as a cache (Hugging Face Hub format), or `None` to use the default location. file_glob: Only files matching this glob will be downloaded to the cache. + adapter_type: Adapter-type slot — `"lora"`, `"alora"`, or `"prompt"`. Returns: - Full path to the local copy of the specified (a)LoRA adapter, suitable for + Full path to the local copy of the specified adapter, suitable for passing to commands that serve the adapter. Raises: - ValueError: If the specified intrinsic adapter cannot be found in the - Hugging Face Hub repository at the expected path. + AdapterNotFoundError: If the specified intrinsic adapter cannot be found in + the Hugging Face Hub repository at the expected path. """ # Third Party import huggingface_hub lora_subdir_name = adapter_subpath( - intrinsic_name, target_model_name, repo_id, alora=alora + intrinsic_name, target_model_name, repo_id, adapter_type=adapter_type ) - # Download just the files for this LoRA if not already present + # Download just the files for this adapter if not already present local_root_path = huggingface_hub.snapshot_download( repo_id=repo_id, allow_patterns=f"{lora_subdir_name}/{file_glob}", @@ -187,12 +201,10 @@ def obtain_lora( # Hugging Face Hub API will happily download nothing. Check whether that happened. if not os.path.exists(lora_dir): - raise ValueError( - f"Intrinsic '{intrinsic_name}' as " - f"{'aLoRA' if alora else 'LoRA'} adapter on base model " - f"'{target_model_name}' not found in " - f"{repo_id} repository on Hugging Face Hub. " - f"Searched for path {lora_subdir_name}/{file_glob}" + raise AdapterNotFoundError( + f"Intrinsic '{intrinsic_name}' as {adapter_type!r} adapter on base " + f"model '{target_model_name}' not found in {repo_id} repository on " + f"Hugging Face Hub. Searched for path {lora_subdir_name}/{file_glob}" ) return lora_dir @@ -204,8 +216,8 @@ def obtain_io_yaml( repo_id: str, /, revision: str = "main", - alora: bool = False, cache_dir: str | None = None, + adapter_type: str = "lora", ) -> pathlib.Path: """Download cached `io.yaml` configuration file for an intrinsic. @@ -217,13 +229,14 @@ def obtain_io_yaml( Args: intrinsic_name: Short name of the intrinsic model, such as `"certainty"`. - target_model_name: Name of the base model for the LoRA or aLoRA adapter. + target_model_name: Name of the base model for the adapter, or `"any"` for a + model-agnostic prompt fallback. repo_id: Hugging Face Hub repository containing a collection of LoRA and/or aLoRA adapters for intrinsics. revision: Git revision of the repository to download from. - alora: If `True`, load the aLoRA version of the intrinsic; otherwise use LoRA. cache_dir: Local directory to use as a cache (Hugging Face Hub format), or `None` to use the default location. + adapter_type: Adapter-type slot — `"lora"`, `"alora"`, or `"prompt"`. Returns: Full path to the local copy of the `io.yaml` file, suitable for passing to @@ -234,8 +247,69 @@ def obtain_io_yaml( target_model_name, repo_id, revision=revision, - alora=alora, cache_dir=cache_dir, file_glob="io.yaml", + adapter_type=adapter_type, ) return lora_dir / "io.yaml" + + +def resolve_prompt_io_yaml( + intrinsic_name: str, + base_model_name: str, + repo_id: str, + /, + revision: str = "main", + cache_dir: str | None = None, +) -> tuple[pathlib.Path, bool]: + """Resolve a prompt fallback's `io.yaml`, preferring a model-specific one. + + Tries the model-specific prompt (`{intrinsic}/{base_model_name}/prompt`) first, + then the model-agnostic fallback (`{intrinsic}/any/prompt`). Advances from the + model-specific to the `any` slot only on the not-found `AdapterNotFoundError`; + any other error (network/auth/bad-revision) from the model-specific fetch + propagates. + + Args: + intrinsic_name: Short name of the intrinsic model, such as `"certainty"`. + base_model_name: Name of the base model to look for a model-specific prompt + under. + repo_id: Hugging Face Hub repository containing the prompt collection. + revision: Git revision of the repository to download from. + cache_dir: Local directory to use as a cache (Hugging Face Hub format), or + `None` to use the default location. + + Returns: + A `(path, used_any)` tuple: the local `io.yaml` path, and `True` if the + model-agnostic `any` fallback was used rather than a model-specific prompt + (callers warn on `True`, since an `any` prompt is sub-par). + + Raises: + AdapterNotFoundError: Neither a model-specific prompt nor the `any` fallback + exists. + """ + try: + return ( + obtain_io_yaml( + intrinsic_name, + base_model_name, + repo_id, + revision=revision, + cache_dir=cache_dir, + adapter_type="prompt", + ), + False, + ) + except AdapterNotFoundError: + pass + return ( + obtain_io_yaml( + intrinsic_name, + "any", + repo_id, + revision=revision, + cache_dir=cache_dir, + adapter_type="prompt", + ), + True, + ) diff --git a/mellea/plugins/hooks/adapter_function.py b/mellea/plugins/hooks/adapter_function.py index b70de03f84..4581289f57 100644 --- a/mellea/plugins/hooks/adapter_function.py +++ b/mellea/plugins/hooks/adapter_function.py @@ -17,7 +17,7 @@ class AdapterFunctionInvocationCompletePayload(MelleaBasePayload): name: Adapter function name (e.g. `"answerability"`). revision: Catalog revision of the adapter, or `None` if unpinned. binding_type: Weight-binding reality the adapter ran under (e.g. - `"local_file"`, `"embedded"`, `"server_mediated"`). + `"local_file"`, `"embedded"`, `"prompt"`, `"server_mediated"`). adapter_type: Adapter mechanism (e.g. `"lora"`, `"alora"`). outcome: `"success"`, `"schema_error"`, or `"error"`. error: The exception raised during invocation, or `None` on success. @@ -27,12 +27,12 @@ class AdapterFunctionInvocationCompletePayload(MelleaBasePayload): aggregating across bindings mixes events with different scope. `"local_file"` covers activate→generate→deactivate only (`AdapterMixin.adapter_scope` closes before parsing runs); - `"embedded"` covers generate+JSON-parse - (`_fire_embedded_invocation_complete` in + `"embedded"` and `"prompt"` cover generate+JSON-parse + (`_afire_invocation_complete` in `mellea.backends.adapters._core`). Neither covers the later contract-level `IOContract` validation in `call_intrinsic` — a contract-mismatch on already-valid JSON records `"success"` for - either binding (tracked in #1611). + any binding (tracked in #1611). """ name: str diff --git a/test/backends/test_adapters/test_adapter_resolution.py b/test/backends/test_adapters/test_adapter_resolution.py new file mode 100644 index 0000000000..7235da1041 --- /dev/null +++ b/test/backends/test_adapters/test_adapter_resolution.py @@ -0,0 +1,179 @@ +# Copyright IBM Corp. All Rights Reserved. +# SPDX-License-Identifier: Apache-2.0 + +"""Unit tests for the availability-aware fallback chain walk. + +Exercises `_register_available_composed_adapter` in isolation: the +`obtain_io_yaml` / `resolve_prompt_io_yaml` boundary is patched and `add_adapter` +is mocked, so each test asserts exactly what the walk registers for a given rung +of the chain, with no network, weights lifecycle, or model load. +""" + +import logging +from unittest.mock import MagicMock, patch + +import pytest +import yaml + +from mellea.backends.adapters._core import LocalFileBinding, PromptBinding +from mellea.backends.adapters.adapter import _register_available_composed_adapter +from mellea.backends.adapters.catalog import AdapterType, fetch_intrinsic_metadata +from mellea.formatters.granite.intrinsics import AdapterNotFoundError + +_ANSWERABILITY_META = fetch_intrinsic_metadata("answerability") +_BASE_MODEL = "granite-4.2-8b" +_INTRINSICS_MOD = "mellea.formatters.granite.intrinsics" + + +def _write_io_yaml(tmp_path): + p = tmp_path / "io.yaml" + p.write_text(yaml.safe_dump({"instruction": "Answer the question."})) + return p + + +def _obtain_io_yaml_stub(*, alora_found: bool, lora_found: bool): + """Build a stub `obtain_io_yaml`; `alora_found`/`lora_found` pick which types resolve. + + A found type returns a path; the rest raise `ValueError` — the not-found + signal the walk advances on. + """ + + def stub(*args, **kwargs): + adapter_type = kwargs.get("adapter_type", "lora") + if adapter_type == "alora" and alora_found: + return "/cache/alora/io.yaml" + if adapter_type == "lora" and lora_found: + return "/cache/lora/io.yaml" + raise AdapterNotFoundError("not found") + + return stub + + +def test_alora_found_registers_local_file_binding(): + with ( + patch( + f"{_INTRINSICS_MOD}.obtain_io_yaml", + side_effect=_obtain_io_yaml_stub(alora_found=True, lora_found=True), + ), + patch(f"{_INTRINSICS_MOD}.resolve_prompt_io_yaml") as mock_prompt, + ): + self_mock = MagicMock() + _register_available_composed_adapter( + self_mock, "answerability", _BASE_MODEL, _ANSWERABILITY_META + ) + self_mock.add_adapter.assert_called_once() + adapter = self_mock.add_adapter.call_args.args[0] + assert isinstance(adapter.weights, LocalFileBinding) + assert adapter.weights.adapter_type is AdapterType.ALORA + assert adapter.identity.adapter_type == "alora" + # aLoRA won, so the prompt rungs are never probed. + mock_prompt.assert_not_called() + + +def test_lora_found_when_no_alora(): + with ( + patch( + f"{_INTRINSICS_MOD}.obtain_io_yaml", + side_effect=_obtain_io_yaml_stub(alora_found=False, lora_found=True), + ), + patch(f"{_INTRINSICS_MOD}.resolve_prompt_io_yaml") as mock_prompt, + ): + self_mock = MagicMock() + _register_available_composed_adapter( + self_mock, "answerability", _BASE_MODEL, _ANSWERABILITY_META + ) + self_mock.add_adapter.assert_called_once() + adapter = self_mock.add_adapter.call_args.args[0] + assert isinstance(adapter.weights, LocalFileBinding) + assert adapter.weights.adapter_type is AdapterType.LORA + mock_prompt.assert_not_called() + + +def test_model_specific_prompt_when_no_trained_adapter(tmp_path, caplog): + io_yaml = _write_io_yaml(tmp_path) + with ( + patch( + f"{_INTRINSICS_MOD}.obtain_io_yaml", + side_effect=_obtain_io_yaml_stub(alora_found=False, lora_found=False), + ), + patch( + f"{_INTRINSICS_MOD}.resolve_prompt_io_yaml", return_value=(io_yaml, False) + ), + ): + with caplog.at_level(logging.WARNING, logger="mellea"): + self_mock = MagicMock() + _register_available_composed_adapter( + self_mock, "answerability", _BASE_MODEL, _ANSWERABILITY_META + ) + self_mock.add_adapter.assert_called_once() + adapter = self_mock.add_adapter.call_args.args[0] + assert isinstance(adapter.weights, PromptBinding) + assert adapter.weights.target_model_name == _BASE_MODEL + assert adapter.identity.adapter_type == "prompt" + # The parsed io.yaml is handed to add_adapter as config=. + assert self_mock.add_adapter.call_args.kwargs["config"] == { + "instruction": "Answer the question." + } + # No "any" warning for a model-specific prompt. + assert not any("model-agnostic 'any'" in r.message for r in caplog.records) + + +def test_any_prompt_warns_and_registers_any_binding(tmp_path, caplog): + io_yaml = _write_io_yaml(tmp_path) + with ( + patch( + f"{_INTRINSICS_MOD}.obtain_io_yaml", + side_effect=_obtain_io_yaml_stub(alora_found=False, lora_found=False), + ), + patch( + f"{_INTRINSICS_MOD}.resolve_prompt_io_yaml", return_value=(io_yaml, True) + ), + ): + with caplog.at_level(logging.WARNING, logger="mellea"): + self_mock = MagicMock() + _register_available_composed_adapter( + self_mock, "answerability", _BASE_MODEL, _ANSWERABILITY_META + ) + self_mock.add_adapter.assert_called_once() + adapter = self_mock.add_adapter.call_args.args[0] + assert isinstance(adapter.weights, PromptBinding) + assert adapter.weights.target_model_name == "any" + assert any("model-agnostic 'any'" in r.message for r in caplog.records) + + +def test_hard_error_aborts_without_fallthrough(): + # A non-ValueError on the first (aLoRA) probe must propagate, not advance to + # LoRA / prompt rungs. + def fail(*args, **kwargs): + raise RuntimeError("network down") + + with ( + patch(f"{_INTRINSICS_MOD}.obtain_io_yaml", side_effect=fail), + patch(f"{_INTRINSICS_MOD}.resolve_prompt_io_yaml") as mock_prompt, + ): + self_mock = MagicMock() + with pytest.raises(RuntimeError): + _register_available_composed_adapter( + self_mock, "answerability", _BASE_MODEL, _ANSWERABILITY_META + ) + mock_prompt.assert_not_called() + self_mock.add_adapter.assert_not_called() + + +def test_nothing_found_raises_value_error(): + with ( + patch( + f"{_INTRINSICS_MOD}.obtain_io_yaml", + side_effect=_obtain_io_yaml_stub(alora_found=False, lora_found=False), + ), + patch( + f"{_INTRINSICS_MOD}.resolve_prompt_io_yaml", + side_effect=AdapterNotFoundError("not found"), + ), + ): + self_mock = MagicMock() + with pytest.raises(ValueError, match=r"No adapter .* or prompt fallback"): + _register_available_composed_adapter( + self_mock, "answerability", _BASE_MODEL, _ANSWERABILITY_META + ) + self_mock.add_adapter.assert_not_called() diff --git a/test/backends/test_adapters/test_adapter_resolution_e2e.py b/test/backends/test_adapters/test_adapter_resolution_e2e.py new file mode 100644 index 0000000000..7e70865a91 --- /dev/null +++ b/test/backends/test_adapters/test_adapter_resolution_e2e.py @@ -0,0 +1,205 @@ +# Copyright IBM Corp. All Rights Reserved. +# SPDX-License-Identifier: Apache-2.0 + +"""End-to-end tests for the adapter resolution chain against a real HF repo. + +Only `io.yaml` files are fetched (`obtain_io_yaml` scopes the download), so these +need no model, GPU, or adapter weights — just Hub read access. They are gated on a +Hugging Face token; the repo is private, so they run locally for a logged-in +developer and skip in CI until the layout lands in a repo CI can read. +""" + +import functools +import logging +from unittest.mock import MagicMock + +import pytest + +from mellea.backends.adapters._core import LocalFileBinding, PromptBinding +from mellea.backends.adapters.adapter import _register_available_composed_adapter +from mellea.backends.adapters.catalog import AdapterType, IntrinsicsCatalogEntry +from mellea.formatters.granite.intrinsics.util import ( + obtain_io_yaml, + resolve_prompt_io_yaml, +) +from test.predicates import require_hf_token + +pytestmark = [pytest.mark.e2e, pytest.mark.huggingface, require_hf_token()] + +# TODO: temporary repo; replace with a public repo once it carries this layout. +_TEST_REPO = "ajbozarth/granitelib-rag-r1.0" +_TEST_SHA = "8bad850fc9855fd655cfd34deb44ddef1597255e" # main @ 2026-09-21 + + +# --- Private-repo access guard (temporary; remove with the TODO above) ----------- +# The fixture repo is private, so these tests xfail rather than hard-fail when it is +# unreadable (CI's public-only token, or another developer's), and run only with +# access. Goes inert once the repo is public. + + +@functools.cache +def _test_repo_readable() -> bool: + """Whether the current HF token can read the test repo (checked once).""" + try: + from huggingface_hub import HfApi + + HfApi().repo_info(_TEST_REPO, revision=_TEST_SHA) + except Exception: + return False + return True + + +@pytest.fixture(autouse=True) +def _xfail_if_test_repo_unreadable(): + """xfail the test when the test repo is unreadable with the current token.""" + if not _test_repo_readable(): + pytest.xfail(f"{_TEST_REPO} not readable with the current token (private)") + + +def _test_entry(capability: str = "answerability") -> IntrinsicsCatalogEntry: + """A catalog entry for `capability`.""" + return IntrinsicsCatalogEntry( + name=capability, + repo_id=_TEST_REPO, + revision=_TEST_SHA, + adapter_types=(AdapterType.ALORA, AdapterType.LORA), + ) + + +# --- Low-level: obtain_io_yaml / resolve_prompt_io_yaml against the test repo ------- + + +def test_trained_io_yaml_resolves(): + """A trained aLoRA adapter's io.yaml resolves (answerability/granite-4.1-3b).""" + path = obtain_io_yaml( + "answerability", + "granite-4.1-3b", + _TEST_REPO, + revision=_TEST_SHA, + adapter_type="alora", + ) + assert path.exists() + assert path.name == "io.yaml" + + +def test_lora_only_io_yaml_resolves(): + """citations is LoRA-only on disk: the aLoRA probe misses, the LoRA resolves. + + The absent type surfaces as the not-found `ValueError` the walk advances on; + the present type resolves normally. + """ + with pytest.raises(ValueError): + obtain_io_yaml( + "citations", + "granite-4.1-3b", + _TEST_REPO, + revision=_TEST_SHA, + adapter_type="alora", + ) + path = obtain_io_yaml( + "citations", + "granite-4.1-3b", + _TEST_REPO, + revision=_TEST_SHA, + adapter_type="lora", + ) + assert path.exists() + assert path.name == "io.yaml" + + +def test_model_specific_prompt_resolves(): + """granite-4.2-8b has no trained adapter, but its model-specific prompt resolves.""" + path, used_any = resolve_prompt_io_yaml( + "answerability", "granite-4.2-8b", _TEST_REPO, revision=_TEST_SHA + ) + assert path.exists() + assert used_any is False + + +def test_any_prompt_fallback_resolves(): + """granite-4.2-3b has nothing of its own, so resolution falls back to `any`.""" + path, used_any = resolve_prompt_io_yaml( + "answerability", "granite-4.2-3b", _TEST_REPO, revision=_TEST_SHA + ) + assert path.exists() + assert used_any is True + + +def test_bad_revision_aborts_not_a_valueerror(): + """A bad revision raises a Hub error, not the not-found ValueError. + + The not-found `ValueError` is the fallthrough signal; a genuine error (here a + nonexistent revision) must be distinguishable so resolution aborts instead of + silently advancing down the chain. + """ + with pytest.raises(Exception) as exc_info: + obtain_io_yaml( + "answerability", + "granite-4.2-8b", + _TEST_REPO, + revision="0" * 40, # nonexistent commit sha + adapter_type="prompt", + ) + assert not isinstance(exc_info.value, ValueError) + + +# --- Higher-level: the resolution walk against the test repo (mock backend) -------- +# +# A MagicMock backend means the walk really probes the test repo but its add_adapter +# is a no-op, so no weights download or model load happens — we assert only what +# the walk *resolves to* for each rung. + + +def test_walk_resolves_trained_adapter(): + """granite-4.1-3b resolves to a LocalFileBinding. + + Both aLoRA and LoRA exist for this model; aLoRA is probed first and wins. + """ + backend = MagicMock() + _register_available_composed_adapter( + backend, "answerability", "granite-4.1-3b", _test_entry() + ) + adapter = backend.add_adapter.call_args.args[0] + assert isinstance(adapter.weights, LocalFileBinding) + assert adapter.weights.adapter_type is AdapterType.ALORA + + +def test_walk_resolves_lora_only_adapter(): + """citations/granite-4.1-3b resolves to a LoRA LocalFileBinding. + + Only `lora/` exists on disk, so the aLoRA probe misses and the walk advances + to LoRA. + """ + backend = MagicMock() + _register_available_composed_adapter( + backend, "citations", "granite-4.1-3b", _test_entry("citations") + ) + adapter = backend.add_adapter.call_args.args[0] + assert isinstance(adapter.weights, LocalFileBinding) + assert adapter.weights.adapter_type is AdapterType.LORA + + +def test_walk_resolves_model_specific_prompt(): + """granite-4.2-8b resolves to a PromptBinding.""" + backend = MagicMock() + _register_available_composed_adapter( + backend, "answerability", "granite-4.2-8b", _test_entry() + ) + backend.add_adapter.assert_called_once() + adapter = backend.add_adapter.call_args.args[0] + assert isinstance(adapter.weights, PromptBinding) + assert adapter.weights.target_model_name == "granite-4.2-8b" + assert isinstance(backend.add_adapter.call_args.kwargs["config"], dict) + + +def test_walk_resolves_any_prompt_with_warning(caplog): + """granite-4.2-3b resolves to the `any` prompt and warns.""" + backend = MagicMock() + with caplog.at_level(logging.WARNING, logger="mellea"): + _register_available_composed_adapter( + backend, "answerability", "granite-4.2-3b", _test_entry() + ) + adapter = backend.add_adapter.call_args.args[0] + assert isinstance(adapter.weights, PromptBinding) + assert adapter.weights.target_model_name == "any" + assert any("model-agnostic 'any'" in r.message for r in caplog.records) diff --git a/test/backends/test_adapters/test_embedded_binding.py b/test/backends/test_adapters/test_embedded_binding.py index b0822e9fa8..44a0c10d5a 100644 --- a/test/backends/test_adapters/test_embedded_binding.py +++ b/test/backends/test_adapters/test_embedded_binding.py @@ -17,7 +17,7 @@ EmbeddedBinding, Identity, ) -from mellea.backends.adapters._core import _fire_embedded_invocation_complete +from mellea.backends.adapters._core import _afire_invocation_complete from mellea.plugins.types import HookType @@ -140,7 +140,7 @@ async def test_apply_activation_does_not_fire_invocation_complete(): # apply_activation only edits the request — the real generate+parse # outcome isn't known yet at this point. Both backends fire # `adapter_function_invocation_complete` later instead, once generation - # and parsing resolve (see _fire_embedded_invocation_complete and its + # and parsing resolve (see _afire_invocation_complete and its # callers in openai.py/huggingface.py, issue #1560). Pin that # apply_activation itself still doesn't fire it, so nobody "fixes" this # back to a hardcoded success outcome guessed at request-mutation time. @@ -161,9 +161,7 @@ async def test_apply_activation_does_not_fire_invocation_complete(): @pytest.mark.parametrize("outcome", ["success", "schema_error", "error"]) -async def test_fire_embedded_invocation_complete_swallows_hook_dispatch_failure( - outcome, -): +async def test_afire_invocation_complete_swallows_hook_dispatch_failure(outcome): """A failing hook dispatch must not mask the real outcome/exception it's reporting.""" pytest.importorskip("cpex", reason="cpex not installed — install mellea[hooks]") @@ -175,8 +173,9 @@ async def test_fire_embedded_invocation_complete_swallows_hook_dispatch_failure( ), ): # Must not raise despite the hook dispatch failing. - await _fire_embedded_invocation_complete( + await _afire_invocation_complete( identity=_identity("answerability"), + binding_type="embedded", outcome=outcome, error=ValueError("boom") if outcome != "success" else None, ) diff --git a/test/backends/test_adapters/test_embedded_integration.py b/test/backends/test_adapters/test_embedded_integration.py index 888eaa73bb..6da36b40ee 100644 --- a/test/backends/test_adapters/test_embedded_integration.py +++ b/test/backends/test_adapters/test_embedded_integration.py @@ -230,7 +230,7 @@ async def test_invocation_complete_fires_error_on_unrelated_exception(): async def test_invocation_complete_fires_error_on_generation_failure(): # A failure in the SDK call itself never reaches granite_formatters_processing — - # avalue() raises it straight off the queue — so _await_embedded_generation + # avalue() raises it straight off the queue — so _await_generation_reporting_error # must fire outcome="error" instead. pytest.importorskip("cpex", reason="cpex not installed — install mellea[hooks]") backend = _backend_with_adapter("alora") diff --git a/test/backends/test_adapters/test_prompt_binding.py b/test/backends/test_adapters/test_prompt_binding.py new file mode 100644 index 0000000000..db2a6c9724 --- /dev/null +++ b/test/backends/test_adapters/test_prompt_binding.py @@ -0,0 +1,62 @@ +# Copyright IBM Corp. All Rights Reserved. +# SPDX-License-Identifier: Apache-2.0 + +"""Unit tests for the weightless PromptBinding and the PROMPT adapter type.""" + +import logging + +import pytest + +from mellea.backends.adapters._core import Identity, PromptBinding +from mellea.backends.adapters.catalog import AdapterType + + +def test_adapter_type_prompt_value(): + assert AdapterType.PROMPT.value == "prompt" + + +def test_identity_accepts_prompt(): + identity = Identity( + name="answerability", adapter_type="prompt", capability="answerability" + ) + assert identity.adapter_type == "prompt" + + +def test_identity_rejects_unknown_type(): + with pytest.raises(ValueError): + Identity(name="answerability", adapter_type="bogus") # type: ignore[arg-type] + + +def test_prompt_binding_qualified_name(): + binding = PromptBinding(name="answerability") + assert binding.qualified_name == "answerability_prompt" + + +def test_prompt_binding_lifecycle_verbs_are_noops(): + binding = PromptBinding(name="answerability") + # Genuine no-ops: return nothing and never raise. + assert binding.prepare() is None + assert binding.deactivate() is None + assert binding.release() is None + + +def test_prompt_binding_activate_logs_at_info(caplog): + binding = PromptBinding(name="answerability", target_model_name="any") + with caplog.at_level(logging.INFO, logger="mellea"): + binding.activate() + assert any( + "Prompt fallback activated" in r.message and "answerability" in r.message + for r in caplog.records + ) + + +def test_prompt_binding_resolved_revision_prefers_explicit(): + binding = PromptBinding(name="answerability", revision="abc123") + assert binding.resolved_revision() == "abc123" + + +def test_prompt_binding_resolved_revision_falls_back_to_catalog(): + # None revision resolves via the catalogue's pinned revision for the name. + binding = PromptBinding(name="answerability", revision=None) + assert isinstance(binding.resolved_revision(), str) + assert binding.resolved_revision() != "" diff --git a/test/backends/test_adapters/test_shims.py b/test/backends/test_adapters/test_shims.py index b781bd7c58..645140b3a9 100644 --- a/test/backends/test_adapters/test_shims.py +++ b/test/backends/test_adapters/test_shims.py @@ -27,6 +27,7 @@ from mellea.backends.adapters._core import Identity, LocalFileBinding from mellea.backends.adapters.adapter import AdapterMixin, _composed_adapter_key from mellea.backends.adapters.catalog import AdapterType, IntrinsicsCatalogEntry +from mellea.formatters.granite.intrinsics import AdapterNotFoundError # --------------------------------------------------------------------------- # Helpers @@ -40,6 +41,19 @@ ) +def _lora_only_io_yaml(*args, **kwargs): + """Stub for `obtain_io_yaml`: only the LoRA slot exists, no network. + + `resolve_adapter` is availability-aware and probes `obtain_io_yaml` before + registering. This makes the aLoRA probe miss and the LoRA probe hit, so the + walk registers a LoRA `LocalFileBinding` — the behaviour these resolve tests + assert — without touching Hugging Face Hub. + """ + if kwargs.get("adapter_type") == "alora": + raise AdapterNotFoundError("no aLoRA adapter in this fixture") + return "/fake/answerability/granite-4.1-3b/lora/io.yaml" + + def _make_intrinsic_adapter(intrinsic_name: str = "answerability") -> IntrinsicAdapter: """Construct IntrinsicAdapter with mocked catalog + config (no HF downloads).""" with ( @@ -613,9 +627,15 @@ def fake_add_adapter(a): AdapterMixin._find_adapter(mock_backend, cap, types) ) - with patch( - "mellea.backends.adapters.adapter.fetch_intrinsic_metadata", - return_value=mock_catalog_entry, + with ( + patch( + "mellea.backends.adapters.adapter.fetch_intrinsic_metadata", + return_value=mock_catalog_entry, + ), + patch( + "mellea.backends.adapters.adapter.intrinsics.obtain_io_yaml", + side_effect=_lora_only_io_yaml, + ), ): result = AdapterMixin.resolve_adapter(mock_backend, "answerability") @@ -657,9 +677,15 @@ def test_resolve_adapter_catalog_alias_returns_registered_adapter(): AdapterMixin._find_adapter(mock_backend, cap, types) ) - with patch( - "mellea.backends.adapters.adapter.fetch_intrinsic_metadata", - return_value=mock_catalog_entry, + with ( + patch( + "mellea.backends.adapters.adapter.fetch_intrinsic_metadata", + return_value=mock_catalog_entry, + ), + patch( + "mellea.backends.adapters.adapter.intrinsics.obtain_io_yaml", + side_effect=_lora_only_io_yaml, + ), ): result = AdapterMixin.resolve_adapter(mock_backend, "guardian-core") @@ -750,6 +776,10 @@ def test_resolve_adapter_survives_reentrant_activation_lock(): "mellea.backends.adapters.adapter.fetch_intrinsic_metadata", return_value=mock_catalog_entry, ), + patch( + "mellea.backends.adapters.adapter.intrinsics.obtain_io_yaml", + side_effect=_lora_only_io_yaml, + ), real_lock, # simulate an already-in-progress caller holding the lock ): result = AdapterMixin.resolve_adapter(mock_backend, "answerability") @@ -797,9 +827,15 @@ def fake_add_adapter(a): AdapterMixin._find_adapter(mock_backend, cap, types) ) - with patch( - "mellea.backends.adapters.adapter.fetch_intrinsic_metadata", - return_value=mock_catalog_entry, + with ( + patch( + "mellea.backends.adapters.adapter.fetch_intrinsic_metadata", + return_value=mock_catalog_entry, + ), + patch( + "mellea.backends.adapters.adapter.intrinsics.obtain_io_yaml", + side_effect=_lora_only_io_yaml, + ), ): AdapterMixin.resolve_adapter(mock_backend, "answerability") diff --git a/test/backends/test_huggingface_unit.py b/test/backends/test_huggingface_unit.py index 429af2803b..730549c9e2 100644 --- a/test/backends/test_huggingface_unit.py +++ b/test/backends/test_huggingface_unit.py @@ -5,6 +5,7 @@ import asyncio import json +import logging import threading import time import warnings @@ -41,7 +42,7 @@ IntrinsicAdapter, ServerMediatedBinding, ) -from mellea.backends.adapters._core import Identity +from mellea.backends.adapters._core import Adapter, Identity, IOContract, PromptBinding from mellea.backends.adapters.adapter import ( EmbeddedIntrinsicAdapter, _ShimWeightsBinding, @@ -706,7 +707,7 @@ async def test_embedded_intrinsic_invocation_complete_fires_error_on_generation_ ): # A failure in the backend's own generation call never reaches # granite_formatters_processing — avalue() raises it straight off the - # queue — so _await_embedded_generation must fire outcome="error" instead. + # queue — so _await_generation_reporting_error must fire outcome="error" instead. pytest.importorskip("cpex", reason="cpex not installed — install mellea[hooks]") backend = _make_intrinsic_backend_stub(stub_backend) backend.processing = AsyncMock(return_value=None) @@ -767,9 +768,9 @@ def failing_generate_with_transformers( async def test_legacy_peft_intrinsic_never_fires_embedded_invocation_complete( stub_backend, ): - # Drives the legacy PEFT path (embedded_identity=None) and pins that it - # never fires the embedded helper — guards the isinstance check that - # gates embedded_identity from silently double-firing. + # Drives the legacy PEFT path (shim weights, not Embedded/Prompt) and pins + # that it never fires invocation-complete — guards the isinstance check that + # gates the emit on adapter.weights from silently double-firing. pytest.importorskip("cpex", reason="cpex not installed — install mellea[hooks]") backend = _make_intrinsic_backend_stub(stub_backend) backend.processing = AsyncMock(return_value=None) @@ -830,6 +831,68 @@ def transform(self, chunk, rewritten): assert HookType.ADAPTER_FUNCTION_INVOCATION_COMPLETE not in fired_hook_types +class _StubPromptContract(IOContract): + """Minimal IOContract; the prompt generation path never reads it.""" + + def build_prompt(self, **kwargs: object): + raise NotImplementedError + + def parse(self, raw: str) -> dict[str, object]: + return {} + + +def _make_prompt_generate_stub(): + """A stub carrying only what `_generate_composed_prompt` touches.""" + model = SimpleNamespace( + active_adapters=MagicMock(return_value=[]), set_adapter=MagicMock() + ) + return SimpleNamespace( + _generation_lock=threading.RLock(), + _model=model, + deactivate_peft_adapter=MagicMock(), + ) + + +def test_generate_composed_prompt_runs_on_clean_base_model(caplog): + """The weightless prompt path clears any PEFT adapter, logs activation, generates.""" + stub = _make_prompt_generate_stub() + adapter = Adapter( + identity=Identity( + name="answerability", adapter_type="prompt", capability="answerability" + ), + io_contract=_StubPromptContract(), + weights=PromptBinding(name="answerability", target_model_name="granite-4.2-8b"), + ) + + with caplog.at_level(logging.INFO, logger="mellea"): + out = LocalHFBackend._generate_composed_prompt( + stub, adapter, lambda *a, **k: "generated", "tok", stub._model, {}, {} + ) + + assert out == "generated" + # No adapter_scope on this path — the base model is cleared before generating. + stub.deactivate_peft_adapter.assert_called_once_with("") + # The soft adapter's activation is surfaced at real-adapter severity. + assert any("Prompt fallback activated" in r.message for r in caplog.records) + + +def test_generate_composed_prompt_rejects_non_prompt_binding(): + """It guards its contract: a non-PromptBinding composed Adapter is a TypeError.""" + stub = _make_prompt_generate_stub() + adapter = Adapter( + identity=Identity( + name="answerability", adapter_type="alora", capability="answerability" + ), + io_contract=_StubPromptContract(), + weights=EmbeddedBinding(), + ) + + with pytest.raises(TypeError, match="PromptBinding"): + LocalHFBackend._generate_composed_prompt( + stub, adapter, lambda *a, **k: "x", "tok", stub._model, {}, {} + ) + + @pytest.mark.asyncio async def test_composed_adapter_drives_generate_from_intrinsic(stub_backend): """A composed `Adapter` (not the `IntrinsicAdapter` shim) drives the full diff --git a/test/backends/test_ollama_intrinsics_unit.py b/test/backends/test_ollama_intrinsics_unit.py index 39893e205c..a46501ff79 100644 --- a/test/backends/test_ollama_intrinsics_unit.py +++ b/test/backends/test_ollama_intrinsics_unit.py @@ -626,10 +626,10 @@ def test_resolve_adapter_falls_back_to_lora_when_alora_unavailable( config_path = tmp_path / "io.yaml" config_path.write_text(json.dumps(_SIMPLE_CONFIG), encoding="utf-8") backend = _make_backend(adapter_models={"citations": _ADAPTER_TAG}) - recorded_alora: list[bool] = [] + recorded_adapter_types: list[str] = [] - def _fake_obtain_io_yaml(*_args, alora: bool, **_kwargs) -> Path: - recorded_alora.append(alora) + def _fake_obtain_io_yaml(*_args, adapter_type: str, **_kwargs) -> Path: + recorded_adapter_types.append(adapter_type) return config_path monkeypatch.setattr( @@ -639,7 +639,7 @@ def _fake_obtain_io_yaml(*_args, alora: bool, **_kwargs) -> Path: adapter = backend.resolve_adapter("citations") - assert recorded_alora == [False] + assert recorded_adapter_types == ["lora"] assert adapter.identity.adapter_type == "lora" diff --git a/test/formatters/granite/test_intrinsics_formatters.py b/test/formatters/granite/test_intrinsics_formatters.py index 697ad27b94..32073b3ff8 100644 --- a/test/formatters/granite/test_intrinsics_formatters.py +++ b/test/formatters/granite/test_intrinsics_formatters.py @@ -723,7 +723,7 @@ def test_run_transformers(yaml_json_combo_with_model, gh_run): cfg.base_model_id, cfg.repo_id, revision=cfg.revision, - alora=cfg.is_alora, + adapter_type="alora" if cfg.is_alora else "lora", ) except requests.exceptions.HTTPError: pytest.xfail("Downloads fail on CI server because repo is private") @@ -889,7 +889,7 @@ def test_run_ollama(yaml_json_combo_for_ollama): cfg.base_model_id, cfg.repo_id, revision=cfg.revision, - alora=cfg.is_alora, + adapter_type="alora" if cfg.is_alora else "lora", ) except requests.exceptions.HTTPError: pytest.xfail("Downloads fail on CI server because repo is private") diff --git a/test/formatters/granite/test_prompt_subpath_unit.py b/test/formatters/granite/test_prompt_subpath_unit.py new file mode 100644 index 0000000000..c118cb82c0 --- /dev/null +++ b/test/formatters/granite/test_prompt_subpath_unit.py @@ -0,0 +1,142 @@ +# Copyright IBM Corp. All Rights Reserved. +# SPDX-License-Identifier: Apache-2.0 + +"""Unit tests for the prompt-fallback path helpers.""" + +import pathlib +from unittest.mock import patch + +import pytest + +from mellea.formatters.granite.intrinsics.constants import OLD_LAYOUT_REPOS +from mellea.formatters.granite.intrinsics.util import ( + AdapterNotFoundError, + adapter_subpath, + resolve_prompt_io_yaml, +) + +_NEW_REPO = "ibm-granite/granitelib-rag-r1.0" + + +def test_adapter_subpath_prompt_new_layout(): + assert ( + adapter_subpath( + "answerability", "granite-4.2-8b", _NEW_REPO, adapter_type="prompt" + ) + == "answerability/granite-4.2-8b/prompt" + ) + + +def test_adapter_subpath_any_prompt_new_layout(): + # "any" is absent from BASE_MODEL_TO_CANONICAL_NAME and passes through as-is, + # occupying the model slot. + assert ( + adapter_subpath("answerability", "any", _NEW_REPO, adapter_type="prompt") + == "answerability/any/prompt" + ) + + +def test_adapter_subpath_prompt_old_layout(): + # Old layout swaps the type slot ahead of the model slot. + old_repo = OLD_LAYOUT_REPOS[0] + assert ( + adapter_subpath( + "answerability", "granite-4.2-8b", old_repo, adapter_type="prompt" + ) + == "answerability/prompt/granite-4.2-8b" + ) + + +def test_adapter_subpath_lora_and_alora_slots(): + # Default slot is lora; alora is passed like any other slot. + assert ( + adapter_subpath("answerability", "granite-4.1-3b", _NEW_REPO) + == "answerability/granite-4.1-3b/lora" + ) + assert ( + adapter_subpath( + "answerability", "granite-4.1-3b", _NEW_REPO, adapter_type="alora" + ) + == "answerability/granite-4.1-3b/alora" + ) + + +def test_resolve_prompt_io_yaml_prefers_model_specific(): + model_path = pathlib.Path("/cache/answerability/granite-4.2-8b/prompt/io.yaml") + with patch( + "mellea.formatters.granite.intrinsics.util.obtain_io_yaml", + return_value=model_path, + ) as mock_obtain: + path, used_any = resolve_prompt_io_yaml( + "answerability", "granite-4.2-8b", _NEW_REPO + ) + assert path == model_path + assert used_any is False + # Only the model-specific probe runs when it succeeds. + mock_obtain.assert_called_once() + assert mock_obtain.call_args.args[1] == "granite-4.2-8b" + assert mock_obtain.call_args.kwargs["adapter_type"] == "prompt" + + +def test_resolve_prompt_io_yaml_falls_back_to_any(): + any_path = pathlib.Path("/cache/answerability/any/prompt/io.yaml") + + def side_effect(intrinsic, model, repo, /, **kwargs): + if model == "any": + return any_path + raise AdapterNotFoundError("not found") + + with patch( + "mellea.formatters.granite.intrinsics.util.obtain_io_yaml", + side_effect=side_effect, + ): + path, used_any = resolve_prompt_io_yaml( + "answerability", "granite-4.2-3b", _NEW_REPO + ) + assert path == any_path + assert used_any is True + + +def test_resolve_prompt_io_yaml_raises_when_neither_exists(): + with patch( + "mellea.formatters.granite.intrinsics.util.obtain_io_yaml", + side_effect=AdapterNotFoundError("not found"), + ): + with pytest.raises(AdapterNotFoundError): + resolve_prompt_io_yaml("answerability", "granite-4.2-3b", _NEW_REPO) + + +def test_resolve_prompt_io_yaml_propagates_non_not_found_error(): + # A non-ValueError from the model-specific probe (e.g. network/auth) must + # abort, not fall through to the "any" slot. + def side_effect(intrinsic, model, repo, /, **kwargs): + if model != "any": + raise RuntimeError("network down") + raise AssertionError("must not reach the 'any' probe after a hard error") + + with patch( + "mellea.formatters.granite.intrinsics.util.obtain_io_yaml", + side_effect=side_effect, + ): + with pytest.raises(RuntimeError): + resolve_prompt_io_yaml("answerability", "granite-4.2-8b", _NEW_REPO) + + +def test_resolve_prompt_io_yaml_propagates_valueerror_subclass(): + # A ValueError *subclass* that is not AdapterNotFoundError (e.g. a Hugging Face + # HFValidationError) from the model-specific probe must propagate, not be + # mistaken for "not found" and fall through to the "any" slot. + class FakeHFValidationError(ValueError): + pass + + def side_effect(intrinsic, model, repo, /, **kwargs): + if model != "any": + raise FakeHFValidationError("malformed repo id") + raise AssertionError("must not reach the 'any' probe after a hard error") + + with patch( + "mellea.formatters.granite.intrinsics.util.obtain_io_yaml", + side_effect=side_effect, + ): + with pytest.raises(FakeHFValidationError): + resolve_prompt_io_yaml("answerability", "granite-4.2-8b", _NEW_REPO) diff --git a/test/predicates.py b/test/predicates.py index 8617275baa..975f5db70e 100644 --- a/test/predicates.py +++ b/test/predicates.py @@ -165,6 +165,34 @@ def test_watsonx_generate(): ... return pytest.mark.skipif(False, reason="") +def require_hf_token(): + """Skip unless a Hugging Face token is available. + + Unlike `require_api_key("HF_TOKEN")`, this accepts a token from *either* the + `HF_TOKEN` environment variable *or* a `hf auth login` session (the token + store `huggingface_hub.get_token()` reads). That makes Hub-auth tests run both + for a locally-logged-in developer and in CI (which supplies `HF_TOKEN`). It + only checks that a token is present, not that it is valid or unexpired. + + Example: + ```python + @require_hf_token() + def test_reads_a_private_repo(): ... + ``` + """ + try: + import huggingface_hub + + has_token = bool(huggingface_hub.get_token()) + except Exception: + has_token = False + if not has_token: + return pytest.mark.skipif( + True, reason="No Hugging Face token (set HF_TOKEN or run `hf auth login`)" + ) + return pytest.mark.skipif(False, reason="") + + # --------------------------------------------------------------------------- # NLTK data # ---------------------------------------------------------------------------