Ingredient normalization and matching

[SaitejaKommi]

Priority

P1

Area

Normalization / Synonyms

Problem

Ingredient search (GET /ingredient/{ingredient}) runs an exact AND match on raw ingredients text. Canadian ingredient labels are noisy, containing bilingual text ('eau / water'), percentages ('milk 2%'), and additive codes ('citric acid E330'). Searching for common ingredient synonyms (e.g. 'soya' vs 'soybean oil', 'chickpeas' vs 'garbanzo beans') results in false negatives.

Why This Matters

Food packaging terminology is non-standardized. Without ingredient normalization and synonym handling, consumers searching for ingredients or allergens encounter false negatives that obscure relevant products.

Current Behaviour

• backend/utils/off_parser.py only strips newlines and selects language in parse_ingredients_text.
• GET /ingredient/{ingredient} in backend/api/routes.py runs a direct query against ingredients.
• backend/search/synonyms_ca.txt contains only 7 basic synonym pairs.
• No structured ingredient tokenization or additive code resolution exists.

Expected Behaviour

• Packaging noise, percentages, and bilingual delimiters are cleaned deterministically.
• Common spelling variants, multilingual terminology, and culinary synonyms are handled (e.g. chickpeas/garbanzo, canola/rapeseed, soy/soya).
• Exact ingredient matching remains supported.
• Negated ingredient searches behave reliably without false substring matches.
• Structured ingredient list populated alongside raw text in search documents.

Proposed Implementation

1. Deterministic Normalization: Build backend/utils/ingredient_normalizer.py to tokenize ingredient text, strip percentages and parentheses, normalize bilingual compound terms (eau/water -> water), and resolve additive codes (e.g. E330 -> citric acid).
2. Expand Canadian Synonyms: Augment backend/search/synonyms_ca.txt with evidence-backed food synonyms from the Canadian catalog.
3. Structured Ingredient Tokens: Populate attributes['ingredients_list'] in SearchDocument with clean, normalized ingredient tokens.

Acceptance Criteria

• Packaging noise, percentages, and bilingual compound phrases are normalized.
• Searching for soy matches soya and soybean oil.
• Searching for chickpeas matches garbanzo beans.
• Exact ingredient matching remains supported.
• Rules are deterministic, maintainable, and covered by tests.
• Existing behaviour is not regressed.
• Appropriate regression tests are added.
• Documentation is updated where necessary.

Implementation Prompt

First inspect the existing implementation and tests before making changes. Understand the current behaviour and identify the smallest appropriate change. Implement the requested functionality without unrelated refactoring. Add or update regression tests. Run the relevant tests, linting/type checks, and verification commands. Do not modify unrelated components.

Inspect `backend/utils/off_parser.py`, `backend/search/mappings.py`, `backend/search/synonyms_ca.py`, and `backend/search/synonyms_ca.txt`.
1. Implement an ingredient text normalizer in `backend/utils/ingredient_normalizer.py` that parses raw ingredients text, handles bilingual delimiters (e.g. 'water / eau'), and removes percentages.
2. Add evidence-backed Canadian ingredient synonym pairs to `backend/search/synonyms_ca.txt` (e.g. chickpeas/garbanzo beans, canola oil/rapeseed oil).
3. Ensure SearchDocumentBuilder incorporates normalized ingredient tokens.
4. Add unit tests in `backend/tests/test_normalizers.py` and `backend/tests/test_synonyms.py` covering bilingual parsing and synonym retrieval.

Verification Plan

Run normalizer and synonym test suites:

pytest backend/tests/test_synonyms.py backend/tests/test_normalizers.py -v
curl "http://127.0.0.1:8000/ingredient/chickpeas"
curl "http://127.0.0.1:8000/ingredient/garbanzo"

Dependencies

None

Maintainer Decision Required

No maintainer decision required.

Out of Scope

• External chemical formula database API lookup.
• Altering user-facing product display of original packaging ingredient declarations.

Relevant Files

• backend/utils/off_parser.py
• backend/utils/ingredient_normalizer.py
• backend/search/synonyms_ca.py
• backend/search/synonyms_ca.txt
• backend/builders/search_document_builder.py
• backend/tests/test_synonyms.py
• backend/tests/test_normalizers.py