Prototype adaptation of AskOFF Search to another country

[SaitejaKommi]

Priority

P2

Area

Ingestion / Proof of Concept

Problem

AskOFF has only ever run on the Canadian dataset. Discrepancies in columns, language fields, and regional dietary marks (e.g. India's vegetarian green dot) in other national datasets have never been empirically tested.

Why This Matters

Prototyping an adaptation against a second national catalog proves whether the country-agnostic abstraction actually works in practice, exposing unforeseen data differences before committing to multi-country support.

Current Behaviour

• backend/adapters/off_adapter.py contains schema introspection via DuckDB, but has only been tested against Canadian Parquet data.
• Reusability of ingestion and parsing for other national datasets is unverified.

Expected Behaviour

• Dataset ingestion works for the selected country (e.g. India or France).
• Country-specific configuration from Backlog ID 19 is used.
• Search and constrained queries work on the prototype catalog.
• Data-quality differences, missing fields, and localized dietary marks are documented.
• No Canada-specific assumptions are hard-coded into the core engine.
• Results are evaluated against a representative query set.

Proposed Implementation

1. Dataset Selection: In consultation with maintainers, obtain a sample export (e.g. 5,000 to 20,000 products) from Open Food Facts India or France.
2. Prototype Adapter: Implement a prototype adapter subclassing BaseAdapter in backend/adapters/prototype_adapter.py.
3. Validate Transformation: Ingest sample products through SearchDocumentBuilder and verify schema validity.
4. Audit Report: Document findings in docs/PROTOTYPE_COUNTRY_ADAPTATION.md detailing reusability metrics, required adjustments, and retrieval accuracy.

Acceptance Criteria

• Prototype adapter ingests sample foreign dataset into SearchDocument without errors.
• Country-specific configuration is used.
• Data-quality differences and schema quirks are documented.
• No Canada-specific assumptions are introduced into core retrieval.
• Canadian functionality remains 100% unaffected.
• Existing behaviour is not regressed.
• Appropriate regression tests are added.
• Documentation is updated where necessary.

Implementation Prompt

First inspect the existing implementation and tests before making changes. Understand the current behaviour and identify the smallest appropriate change. Implement the requested functionality without unrelated refactoring. Add or update regression tests. Run the relevant tests, linting/type checks, and verification commands. Do not modify unrelated components.

Inspect `backend/adapters/base.py`, `backend/adapters/off_adapter.py`, and `backend/builders/search_document_builder.py`.
Create a proof-of-concept country adaptation for a second Open Food Facts regional dataset (e.g. India or France):
1. Obtain or inspect a sample Parquet/CSV dataset from Open Food Facts India or France (e.g. 1,000 - 5,000 products).
2. Create a prototype adapter in `backend/adapters/prototype_adapter.py` implementing `extract_raw_products()`.
3. Verify that SearchDocumentBuilder builds valid SearchDocuments with proper nutrition parsing and flags.
4. Document findings, schema differences, and pipeline reusability in `docs/PROTOTYPE_COUNTRY_ADAPTATION.md`.
5. Add unit tests in `backend/tests/test_prototype_adapter.py` verifying the ingestion of sample products.

Verification Plan

Run prototype tests and verify Canadian suite stability:

pytest backend/tests/test_prototype_adapter.py -v
pytest backend/tests/ -v

Dependencies

Backlog IDs 18 and 19

Maintainer Decision Required

A maintainer decision is required regarding the selection of the target country for the proof of concept (e.g. India vs France).

Out of Scope

• Full production deployment of another country catalog.
• Building full regional synonym dictionaries for all foreign languages.

Relevant Files

• backend/adapters/base.py
• backend/adapters/off_adapter.py
• backend/builders/search_document_builder.py
• backend/tests/test_prototype_adapter.py
• docs/PROTOTYPE_COUNTRY_ADAPTATION.md