diff --git a/examples/sie-hugging-face-mteb-semantic-search/README.md b/examples/sie-hugging-face-mteb-semantic-search/README.md index dbc338544..a6f314ba2 100644 --- a/examples/sie-hugging-face-mteb-semantic-search/README.md +++ b/examples/sie-hugging-face-mteb-semantic-search/README.md @@ -20,8 +20,10 @@ the SIE model IDs. **Two modes.** *Zero-setup mode* runs a local text ranker against the bundled demo catalog (no SIE, no API keys required, good for kicking -the tires). *Full mode* uses your SIE endpoint for embeddings and -OpenRouter for LLM-generated descriptions (the production path). Copy +the tires). *Full mode* uses your SIE endpoint for embeddings and a +configurable LLM provider — OpenRouter by default, or +[OrcaRouter](https://www.orcarouter.ai) — for LLM-generated +descriptions (the production path). Copy `backend/.env.example` to `backend/.env` and fill in only the keys for the mode you want. The sections below walk through both. @@ -53,7 +55,7 @@ the mode you want. The sections below walk through both. - SQLite in `backend/data/sqlite/sie.db` with tables `storage_ids` and `models`. - ChromaDB in `backend/data/chroma/` as the local vector store. - Superlinked Inference Engine (SIE) produces embeddings for short and long descriptions. - - OpenRouter generates descriptions from HF metadata + README + MTEB scores. + - The configured LLM provider (OpenRouter by default, or OrcaRouter) generates descriptions from HF metadata + README + MTEB scores. - **Frontend**: TypeScript + React app in `frontend/` that calls the backend APIs for search, browse, and model details. ```mermaid @@ -63,7 +65,7 @@ flowchart LR SQL[(SQLite
models, storage_ids)] Chroma[(ChromaDB
short + long vectors)] SIE[SIE
embeddings] - OR[OpenRouter
description LLM] + OR[LLM provider
OpenRouter / OrcaRouter] HF[HuggingFace
metadata + README] MTEB[MTEB cache
benchmark scores] @@ -80,7 +82,7 @@ flowchart LR ## Try It Locally First -You can run the project without a SIE endpoint, OpenRouter key, or Hugging Face token. +You can run the project without a SIE endpoint, LLM API key, or Hugging Face token. The repo includes a small bundled demo catalog plus a local text-ranking fallback. ```bash @@ -105,7 +107,7 @@ What works in local demo mode: What still needs live services: - downloading fresh Hugging Face model metadata -- generating descriptions through OpenRouter +- generating descriptions through the configured LLM provider - vector search and Chroma reindexing through a real SIE endpoint --- @@ -132,7 +134,7 @@ sie-hugging-face-mteb-semantic-search/ │ ├── app/ │ │ ├── api/routes/ # FastAPI routers: models, generate, search, chroma │ │ ├── db/ # SQLAlchemy models, session, migrations -│ │ ├── services/ # chroma, fallback search, llm, openrouter, sie_chroma +│ │ ├── services/ # chroma, fallback search, llm, openrouter, orcarouter, sie_chroma │ │ ├── prompts/ # description prompt templates (.md) │ │ ├── config.py # pydantic-settings, reads backend/.env │ │ └── main.py # FastAPI app factory @@ -157,7 +159,7 @@ sie-hugging-face-mteb-semantic-search/ - **Python 3.12** and `pip`. - **Node.js 18+** and `npm`. -- An **OpenRouter** API key if you want to generate new descriptions. +- An **OpenRouter** or **OrcaRouter** API key if you want to generate new descriptions (set `LLM_PROVIDER` to choose the provider). - A running **SIE** endpoint if you want live vector indexing and embedding search. `SIE_API_KEY` is optional and only needed for managed/auth-enabled clusters. - Optional: a **Hugging Face** token, useful for higher rate limits. @@ -189,9 +191,12 @@ See `backend/app/config.py` for the full list; the important keys are: | Variable | Default | Purpose | |-----------------------|--------------------------------------|--------------------------------------------| | `HF_TOKEN` | _(empty)_ | Optional, raises HuggingFace rate limits | +| `LLM_PROVIDER` | `openrouter` | Description LLM provider: `openrouter`, `orcarouter`, or `openai` | | `OPENROUTER_API_KEY` | _(empty, required for generation)_ | Auth for OpenRouter description calls | -| `OPENROUTER_MODEL` | `google/gemini-3.1-pro-preview` | Default LLM used by CLI + UI | -| `LLM_MAX_PARALLEL` | `20` | Max in-flight OpenRouter calls | +| `OPENROUTER_MODEL` | `google/gemini-3.1-pro-preview` | Default LLM used by CLI + UI when provider is OpenRouter | +| `ORCAROUTER_API_KEY` | _(empty, required for generation)_ | Auth for OrcaRouter description calls | +| `ORCAROUTER_MODEL` | `google/gemini-2.5-flash` | Default LLM used by CLI + UI when provider is OrcaRouter | +| `LLM_MAX_PARALLEL` | `20` | Max in-flight LLM provider calls | | `SIE_API_ENDPOINT` | _(empty, required for embeddings)_ | URL of the SIE server | | `SIE_API_KEY` | _(empty)_ | Optional bearer token for managed/auth-enabled SIE clusters | | `SIE_EMBED_MODEL` | `NovaSearch/stella_en_400M_v5` | Embedding model registered on SIE | @@ -199,9 +204,10 @@ See `backend/app/config.py` for the full list; the important keys are: | `SQLITE_PATH` | `data/sqlite/sie.db` | Local SQLite database path | | `CHROMA_PATH` | `data/chroma` | Local ChromaDB directory | -Minimal `backend/.env` for live services: +Minimal `backend/.env` for live services (OpenRouter): ```env +LLM_PROVIDER=openrouter OPENROUTER_API_KEY=sk-or-... SIE_API_ENDPOINT=https://your-sie-host # Optional: only needed for managed/auth-enabled SIE clusters. @@ -209,6 +215,18 @@ SIE_API_KEY= HF_TOKEN=hf_... # optional ``` +To use OrcaRouter instead, set `LLM_PROVIDER=orcarouter` and add the +OrcaRouter key: + +```env +LLM_PROVIDER=orcarouter +ORCAROUTER_API_KEY=sk-orca-... +SIE_API_ENDPOINT=https://your-sie-host +# Optional: only needed for managed/auth-enabled SIE clusters. +SIE_API_KEY= +HF_TOKEN=hf_... # optional +``` + --- ## How to use @@ -286,7 +304,7 @@ for downloaded models (see [Operations notes](#operations-notes)). Runs the same pipeline as the web UI *Generate Descriptions* buttons: 1. Prepare 6K prompt (HF metadata + live README + MTEB summary). -2. Generate 6K detailed description via OpenRouter. +2. Generate 6K detailed description via the configured LLM provider. 3. Generate 2K long description from the 6K output. 4. Generate 200-char short description from the 6K output. 5. Save short + long to SQLite. @@ -499,9 +517,9 @@ is what actually reclaims disk. The CLI and UI both follow the same six-step pipeline: 1. Render the **6K detailed** prompt from model JSON, live README (4K chars max), and MTEB summary. -2. Call OpenRouter → **6K detailed description** (not persisted). -3. Call OpenRouter with the 6K text → **2K long description**. -4. Call OpenRouter with the 6K text → **200-char short description**. +2. Call the configured LLM provider → **6K detailed description** (not persisted). +3. Call the provider with the 6K text → **2K long description**. +4. Call the provider with the 6K text → **200-char short description**. 5. Save short + long into the `models` table. 6. Upsert short + long embeddings into ChromaDB via SIE. diff --git a/examples/sie-hugging-face-mteb-semantic-search/backend/.env.example b/examples/sie-hugging-face-mteb-semantic-search/backend/.env.example index 190e67c0a..4e109a504 100644 --- a/examples/sie-hugging-face-mteb-semantic-search/backend/.env.example +++ b/examples/sie-hugging-face-mteb-semantic-search/backend/.env.example @@ -11,10 +11,20 @@ HF_TOKEN= OPENAI_API_KEY= OPENAI_MODEL=gpt-4o-mini +# ── LLM provider for description generation ─────────────────────────── +# One of: openrouter (default), orcarouter, openai. +LLM_PROVIDER=openrouter + # ── OpenRouter (full mode) ──────────────────────────────────────────── # Used for LLM-generated descriptions in full mode. OPENROUTER_API_KEY= OPENROUTER_MODEL=google/gemini-3.1-pro-preview + +# ── OrcaRouter (full mode) ──────────────────────────────────────────── +# Set LLM_PROVIDER=orcarouter to use OrcaRouter for description generation. +ORCAROUTER_API_KEY= +ORCAROUTER_MODEL=google/gemini-2.5-flash + LLM_MAX_PARALLEL=20 # ── Superlinked Inference Engine ────────────────────────────────────── diff --git a/examples/sie-hugging-face-mteb-semantic-search/backend/app/api/routes/generate.py b/examples/sie-hugging-face-mteb-semantic-search/backend/app/api/routes/generate.py index acdbc92ba..b14db8948 100644 --- a/examples/sie-hugging-face-mteb-semantic-search/backend/app/api/routes/generate.py +++ b/examples/sie-hugging-face-mteb-semantic-search/backend/app/api/routes/generate.py @@ -9,8 +9,8 @@ from app.db import models as db_models from app.db.session import get_db from app.prompts import load_prompt -from app.services.openrouter import generate_text from app.services.chroma import upsert_embedding +from app.services.llm import generate_text logger = logging.getLogger(__name__) @@ -139,7 +139,7 @@ def render_prompt(payload: RenderRequest, db: Session = Depends(get_db)): class GenerateDetailedRequest(BaseModel): prompt_text: str = Field(..., description="The (possibly edited) detailed prompt") model: Optional[str] = Field( - None, description="OpenRouter model name (uses default from config if omitted)" + None, description="LLM model name (uses the provider default from config if omitted)" ) @@ -150,7 +150,7 @@ class GenerateFromDetailedRequest(BaseModel): ..., description="The 6K detailed description output" ) model: Optional[str] = Field( - None, description="OpenRouter model name (uses default from config if omitted)" + None, description="LLM model name (uses the provider default from config if omitted)" ) diff --git a/examples/sie-hugging-face-mteb-semantic-search/backend/app/config.py b/examples/sie-hugging-face-mteb-semantic-search/backend/app/config.py index 86d619a4f..9c04132f8 100644 --- a/examples/sie-hugging-face-mteb-semantic-search/backend/app/config.py +++ b/examples/sie-hugging-face-mteb-semantic-search/backend/app/config.py @@ -15,9 +15,17 @@ class Settings(BaseSettings): openai_api_key: str = "" openai_model: str = "gpt-4o-mini" + # LLM provider used for description generation: "openrouter" (default) or "orcarouter" + llm_provider: str = "openrouter" + # OpenRouter openrouter_api_key: str = "" openrouter_model: str = "google/gemini-3.1-pro-preview" + + # OrcaRouter + orcarouter_api_key: str = "" + orcarouter_model: str = "google/gemini-2.5-flash" + llm_max_parallel: int = 20 # Superlinked Inference Engine @@ -35,6 +43,13 @@ def database_url(self) -> str: db_path = self.sqlite_path.resolve() return f"sqlite:///{db_path}" + @property + def llm_model(self) -> str: + """Default LLM model for the active provider.""" + if self.llm_provider.strip().lower() == "orcarouter": + return self.orcarouter_model + return self.openrouter_model + model_config = {"env_file": ".env", "env_file_encoding": "utf-8"} diff --git a/examples/sie-hugging-face-mteb-semantic-search/backend/app/services/llm.py b/examples/sie-hugging-face-mteb-semantic-search/backend/app/services/llm.py index d95ef77c9..92b9c82a0 100644 --- a/examples/sie-hugging-face-mteb-semantic-search/backend/app/services/llm.py +++ b/examples/sie-hugging-face-mteb-semantic-search/backend/app/services/llm.py @@ -1,15 +1,18 @@ +import asyncio import logging +from typing import Optional from openai import OpenAI from app.config import settings +from app.services import orcarouter, openrouter logger = logging.getLogger(__name__) _client: OpenAI | None = None -def _get_client() -> OpenAI: +def _get_openai_client() -> OpenAI: global _client if _client is None: if not settings.openai_api_key: @@ -20,9 +23,9 @@ def _get_client() -> OpenAI: return _client -def generate_text(prompt: str, max_tokens: int = 4096) -> str: +def _generate_openai(prompt: str, max_tokens: int = 4096) -> str: """Send a prompt to OpenAI and return the assistant's response text.""" - client = _get_client() + client = _get_openai_client() logger.info( "Calling OpenAI %s (prompt length: %d chars, max_tokens: %d)", settings.openai_model, @@ -38,3 +41,44 @@ def generate_text(prompt: str, max_tokens: int = 4096) -> str: text = response.choices[0].message.content or "" logger.info("OpenAI response: %d chars", len(text)) return text.strip() + + +def generate_text( + prompt: str, + max_tokens: int = 4096, + model: Optional[str] = None, +) -> str: + """Generate text with the configured provider (defaults to OpenRouter). + + Supported providers (LLM_PROVIDER): ``openrouter`` (default), + ``orcarouter``, and ``openai``. + """ + provider = settings.llm_provider.strip().lower() + if provider == "orcarouter": + return orcarouter.generate_text(prompt, max_tokens=max_tokens, model=model) + if provider == "openai": + return _generate_openai(prompt, max_tokens=max_tokens) + return openrouter.generate_text(prompt, max_tokens=max_tokens, model=model) + + +async def generate_text_async( + prompt: str, + max_tokens: int = 4096, + model: Optional[str] = None, + semaphore: asyncio.Semaphore | None = None, +) -> str: + """Async generate_text with the configured provider (defaults to OpenRouter). + + Supported providers (LLM_PROVIDER): ``openrouter`` (default), + ``orcarouter``, and ``openai`` (sync fallback). + """ + provider = settings.llm_provider.strip().lower() + if provider == "orcarouter": + return await orcarouter.generate_text_async( + prompt, max_tokens=max_tokens, model=model, semaphore=semaphore + ) + if provider == "openai": + return await asyncio.to_thread(_generate_openai, prompt, max_tokens) + return await openrouter.generate_text_async( + prompt, max_tokens=max_tokens, model=model, semaphore=semaphore + ) diff --git a/examples/sie-hugging-face-mteb-semantic-search/backend/app/services/openrouter.py b/examples/sie-hugging-face-mteb-semantic-search/backend/app/services/openrouter.py index b4eb1b863..770337bcd 100644 --- a/examples/sie-hugging-face-mteb-semantic-search/backend/app/services/openrouter.py +++ b/examples/sie-hugging-face-mteb-semantic-search/backend/app/services/openrouter.py @@ -71,7 +71,7 @@ def generate_text( ) -> str: """Send a prompt to OpenRouter and return the assistant's response text.""" client = _get_client() - model_name = model or settings.openrouter_model + model_name = model or settings.llm_model logger.info( "Calling OpenRouter model=%s (prompt length: %d chars, max_tokens: %d)", @@ -97,7 +97,7 @@ async def generate_text_async( ) -> str: """Async version of generate_text with automatic retry on 429 rate-limit.""" client = _get_async_client() - model_name = model or settings.openrouter_model + model_name = model or settings.llm_model async with semaphore or asyncio.Semaphore(1): for attempt in range(_RETRY_MAX): diff --git a/examples/sie-hugging-face-mteb-semantic-search/backend/app/services/orcarouter.py b/examples/sie-hugging-face-mteb-semantic-search/backend/app/services/orcarouter.py new file mode 100644 index 000000000..cdabd74ee --- /dev/null +++ b/examples/sie-hugging-face-mteb-semantic-search/backend/app/services/orcarouter.py @@ -0,0 +1,132 @@ +import asyncio +import logging +from typing import Optional + +from openai import APIStatusError, AsyncOpenAI, OpenAI + +from app.config import settings + +logger = logging.getLogger(__name__) + +_ORCAROUTER_BASE_URL = "https://api.orcarouter.ai/v1" + +_client: OpenAI | None = None +_async_client: AsyncOpenAI | None = None + +_RETRY_MAX = 5 +_RETRY_BASE_DELAY = 1.0 # seconds; doubles each attempt + + +def _get_client() -> OpenAI: + global _client + if _client is None: + if not settings.orcarouter_api_key: + raise RuntimeError( + "ORCAROUTER_API_KEY is not set. Add it to backend/.env" + ) + _client = OpenAI( + api_key=settings.orcarouter_api_key, + base_url=_ORCAROUTER_BASE_URL, + ) + return _client + + +def _get_async_client() -> AsyncOpenAI: + global _async_client + if _async_client is None: + if not settings.orcarouter_api_key: + raise RuntimeError( + "ORCAROUTER_API_KEY is not set. Add it to backend/.env" + ) + _async_client = AsyncOpenAI( + api_key=settings.orcarouter_api_key, + base_url=_ORCAROUTER_BASE_URL, + ) + return _async_client + + +def _log_usage(response) -> None: + text = response.choices[0].message.content or "" + usage = response.usage + if usage: + details = getattr(usage, "completion_tokens_details", None) + reasoning = getattr(details, "reasoning_tokens", None) if details else None + logger.info( + "OrcaRouter usage: prompt_tokens=%s, completion_tokens=%s, " + "reasoning_tokens=%s, content_chars=%d, finish_reason=%s", + usage.prompt_tokens, + usage.completion_tokens, + reasoning, + len(text), + response.choices[0].finish_reason, + ) + else: + logger.info("OrcaRouter response: %d chars", len(text)) + + +def generate_text( + prompt: str, + max_tokens: int = 4096, + model: Optional[str] = None, +) -> str: + """Send a prompt to OrcaRouter and return the assistant's response text.""" + client = _get_client() + model_name = model or settings.orcarouter_model + + logger.info( + "Calling OrcaRouter model=%s (prompt length: %d chars, max_tokens: %d)", + model_name, + len(prompt), + max_tokens, + ) + response = client.chat.completions.create( + model=model_name, + messages=[{"role": "user", "content": prompt}], + max_tokens=max_tokens, + temperature=0.4, + ) + _log_usage(response) + return (response.choices[0].message.content or "").strip() + + +async def generate_text_async( + prompt: str, + max_tokens: int = 4096, + model: Optional[str] = None, + semaphore: asyncio.Semaphore | None = None, +) -> str: + """Async version of generate_text with automatic retry on 429 rate-limit.""" + client = _get_async_client() + model_name = model or settings.orcarouter_model + + async with semaphore or asyncio.Semaphore(1): + for attempt in range(_RETRY_MAX): + logger.info( + "Calling OrcaRouter (async) model=%s (prompt length: %d chars, max_tokens: %d)", + model_name, + len(prompt), + max_tokens, + ) + try: + response = await client.chat.completions.create( + model=model_name, + messages=[{"role": "user", "content": prompt}], + max_tokens=max_tokens, + temperature=0.4, + ) + _log_usage(response) + return (response.choices[0].message.content or "").strip() + except APIStatusError as exc: + if exc.status_code == 429 and attempt < _RETRY_MAX - 1: + delay = _RETRY_BASE_DELAY * (2 ** attempt) + logger.warning( + "Rate-limited (429); retrying in %.1fs (attempt %d/%d)", + delay, + attempt + 1, + _RETRY_MAX, + ) + await asyncio.sleep(delay) + continue + raise + + raise RuntimeError("Unreachable: retry loop exited without return or raise") diff --git a/examples/sie-hugging-face-mteb-semantic-search/backend/cli_generate.py b/examples/sie-hugging-face-mteb-semantic-search/backend/cli_generate.py index 61ed8ce04..801b4cff6 100644 --- a/examples/sie-hugging-face-mteb-semantic-search/backend/cli_generate.py +++ b/examples/sie-hugging-face-mteb-semantic-search/backend/cli_generate.py @@ -2,7 +2,7 @@ Runs the same pipeline as the web UI Generate Descriptions buttons: 1. Prepare 6K prompt (model metadata + README + MTEB summary) - 2. Generate 6K detailed description via OpenRouter + 2. Generate 6K detailed description via the configured LLM provider 3. Generate 2K long description from the 6K output 4. Generate 200-char short description from the 6K output 5. Save both descriptions to the database @@ -30,7 +30,7 @@ from app.db.session import SessionLocal from app.prompts import load_prompt from app.services.chroma import reindex_all, upsert_embedding -from app.services.openrouter import generate_text, generate_text_async +from app.services.llm import generate_text, generate_text_async logging.basicConfig( level=logging.INFO, @@ -312,7 +312,7 @@ def main(): parser.add_argument( "--model", default=None, - help=f"OpenRouter model name (default: {settings.openrouter_model})", + help=f"LLM model name (default: {settings.llm_model})", ) parser.add_argument( "--parallel", @@ -400,7 +400,7 @@ def main(): "Processing %d model(s) in storage '%s' with model '%s' (parallel=%d)", len(models), args.storage_id, - args.model or settings.openrouter_model, + args.model or settings.llm_model, args.parallel, ) diff --git a/examples/sie-hugging-face-mteb-semantic-search/frontend/frontend.md b/examples/sie-hugging-face-mteb-semantic-search/frontend/frontend.md index 3f8cb0765..f7022ae23 100644 --- a/examples/sie-hugging-face-mteb-semantic-search/frontend/frontend.md +++ b/examples/sie-hugging-face-mteb-semantic-search/frontend/frontend.md @@ -26,7 +26,7 @@ The following UI elements are there - text area 2K description - text area 200 description ##### Functionality Button Generate 6K Description -The prompt text area has already variables replaced. So if the button is pressed, an openrouter model is called with the prompt and the results are filled into the 6K description. This field is not saved. +The prompt text area has already variables replaced. So if the button is pressed, the configured LLM provider model is called with the prompt and the results are filled into the 6K description. This field is not saved. ##### Functionality Button Generate 2K Description Based on the 6K description a 2K description is generated and filled into the corresponding text area. It is not yet saved. The corresponding prompt is used for this (find good prompt even change prompt if needed) diff --git a/examples/sie-hugging-face-mteb-semantic-search/frontend/src/App.tsx b/examples/sie-hugging-face-mteb-semantic-search/frontend/src/App.tsx index 987ec46de..469e60233 100644 --- a/examples/sie-hugging-face-mteb-semantic-search/frontend/src/App.tsx +++ b/examples/sie-hugging-face-mteb-semantic-search/frontend/src/App.tsx @@ -383,7 +383,7 @@ const GenerateModal: React.FC = ({
{

Describe what the model should do and find the best matching embedding models by semantic similarity. Use storage {DEMO_STORAGE_ID} to - try the bundled local demo without SIE or OpenRouter credentials. + try the bundled local demo without SIE or LLM credentials.

@@ -872,7 +872,7 @@ const SearchWithRerankingView: React.FC = () => { First we find candidates by short-description similarity, then rerank them by long-description similarity for higher-quality results. Use storage{" "} {DEMO_STORAGE_ID} to try the bundled local demo without SIE or - OpenRouter credentials. + LLM credentials.