Integrate an SLM/LLM into AskOFF Bot

[SaitejaKommi]

Priority

P1

Area

Conversational Bot / LLM

Problem

The conversational assistant in OffBotWidget.tsx and assistantService.ts currently runs purely on hardcoded TypeScript regex rules in the frontend. There is no backend conversational endpoint. The core architectural principle must be strictly preserved: Do NOT replace deterministic search with an LLM. The LLM must sit on top of deterministic retrieval to explain, summarize, and answer questions grounded strictly in retrieved Golden Product Records.

Why This Matters

Shoppers frequently have natural-language questions about product comparisons, allergen safety, and nutritional trade-offs. Using an SLM/LLM strictly grounded in retrieved Golden Records allows AskOFF to provide rich conversational assistance without sacrificing the speed, reliability, and precision of its deterministic search engine.

Current Behaviour

• frontend/src/components/OffBotWidget.tsx renders the chat UI.
• frontend/src/api/assistantService.ts defines hardcoded client-side question handlers.
• The FastAPI backend exposes search endpoints, but has no /bot/chat endpoint.
• Core search routes currently operate with zero LLM dependency.

Expected Behaviour

• Architecture strictly separates deterministic retrieval from conversational generation:

  User Question
    ↓
  Deterministic Query Understanding
    ↓
  Deterministic Search / OpenSearch
    ↓
  Structured Golden Product Records
    ↓
  SLM / LLM
    ↓
  Grounded Conversational Response
  

• The LLM does NOT replace the deterministic search engine.
• Retrieved product facts are provided as structured context to the model.
• Core search (GET /search) maintains sub-50ms execution with zero runtime LLM dependency.
• If the LLM is unavailable, the API degrades safely to structured product summaries.

Proposed Implementation

1. Backend Conversational Endpoint: Implement POST /bot/chat in FastAPI taking a question and optional active product context or search filters.
2. Deterministic Context Retrieval: Use deterministic query understanding and OpenSearch to retrieve top 1-5 candidate Golden Product Records, assembling structured JSON attributes as context.
3. Pluggable LLM Provider: Implement an abstract LLMProvider interface in backend/bot/provider.py supporting local SLMs (e.g. Ollama Phi-3 / Llama-3-8B) and external API providers.
4. Strict Failure Isolation: Ensure core /search endpoints have zero dependency on the LLM layer. If the LLM times out or fails, return retrieved products directly with an informative fallback message.

Acceptance Criteria

• Architecture strictly separates retrieval from generation.
• Retrieved product information is explicitly available to the model.
• Unsupported claims are minimized through grounding prompts.
• Core search (GET /search) operates with zero LLM dependency and sub-50ms latency.
• Model failures degrade safely to structured product cards.
• Maintainer approval is obtained before committing to a production runtime.
• Existing behaviour is not regressed.
• Appropriate regression tests are added.
• Documentation is updated where necessary.

Implementation Prompt

First inspect the existing implementation and tests before making changes. Understand the current behaviour and identify the smallest appropriate change. Implement the requested functionality without unrelated refactoring. Add or update regression tests. Run the relevant tests, linting/type checks, and verification commands. Do not modify unrelated components.

Inspect `frontend/src/api/assistantService.ts`, `backend/api/routes.py`, `backend/retrieval/search_engine.py`, and `backend/models/search_document.py`.
Create a backend conversational layer following AskOFF's strict architecture (deterministic retrieval -> structured context -> LLM):
1. In `backend/bot/provider.py`, create an abstract LLMProvider interface with a MockLLMProvider and an Ollama/OpenAI-compatible client.
2. In `backend/bot/service.py`, implement the pipeline: take a user prompt, retrieve top candidate products via SearchEngine, format structured context, and invoke the provider.
3. Expose POST /bot/chat in `backend/api/routes.py` with proper Pydantic schemas.
4. Ensure that if the LLM provider fails or times out, the endpoint returns retrieved product cards with a fallback message. Core /search must remain 100% independent.
5. Add unit tests in `backend/tests/test_bot.py`.

Verification Plan

Run bot unit tests and test conversational endpoint:

pytest backend/tests/test_bot.py -v
curl -X POST "http://127.0.0.1:8000/bot/chat" -H "Content-Type: application/json" -d '{"message": "Why is Kraft Peanut Butter high in protein?", "barcode": "0068100084124"}'

Dependencies

Backlog IDs 2 and 6

Maintainer Decision Required

A maintainer decision is required regarding the choice of default SLM/LLM runtime (local Ollama Phi-3/Llama-3-8B vs hosted cloud API), compute requirements, and deployment model before committing to a production runtime.

Out of Scope

• Replacing OpenSearch BM25 retrieval with an LLM agent.
• Fine-tuning a proprietary foundation model from scratch.

Relevant Files

• frontend/src/api/assistantService.ts
• frontend/src/components/OffBotWidget.tsx
• backend/api/routes.py
• backend/retrieval/search_engine.py
• backend/bot/provider.py
• backend/bot/service.py
• backend/tests/test_bot.py