Improve deterministic dietary tags

[SaitejaKommi]

Priority

P1

Area

Ingestion / Data Quality

Problem

Dietary flags in SearchDocumentBuilder contain critical correctness bugs: is_palm_oil_free = 'palm oil' not in ingredients_lower causes ~25,000 products with missing ingredients to evaluate as palm-oil-free; negation phrases like 'not suitable for vegans' match 'vegan' and mark items as vegan; animal ingredients (gelatin, lard) are not checked; and there is no three-state logic (True, False, Unknown/None).

Why This Matters

Dietary attributes directly affect consumer health and ethical compliance. In food search, treating unrecorded ingredients as positive proof of compliance is dangerous. Classifying an unlabelled product as palm-oil-free or vegan compromises Open Food Facts' data integrity.

Current Behaviour

• backend/builders/search_document_builder.py precomputes boolean flags for is_organic, is_vegan, is_vegetarian, is_palm_oil_free, is_high_protein, is_low_sugar, is_low_sodium, is_gluten_free, is_lactose_free.
• Products with empty ingredients_text are classified as is_palm_oil_free: True.
• Negated phrases cause false-positive vegan/vegetarian tags.
• Open Food Facts structured ingredients_analysis_tags (en:vegan, en:non-vegan) are ignored in favor of raw substring searches.

Expected Behaviour

• Rules are explicit, deterministic, and documented.
• Three-state logic is used: known positive (True), known negative (False), unknown/missing (None).
• Missing data is never interpreted as a positive attribute (products with missing ingredients evaluate to None, not True).
• Disqualifying ingredients (animal tokens for vegan, palm oil for palm-oil-free) correctly prevent positive classification.
• Negation phrases ('not suitable for vegans') do not trigger false positive flags.
• Canadian low-calorie standard (<= 40 kcal / 100g) is supported.
• Do not use LLM inference for this issue.

Proposed Implementation

1. Three-State Deterministic Logic:
   • True: Explicitly verified via official tag (en:vegan, en:palm-oil-free) or confirmed ingredient analysis.
   • False: Detected disqualifying ingredient (palm oil present, animal ingredient present, or en:non-vegan tag).
   • None (Unknown): Ingredients unrecorded or analysis status unknown. Never assume unrecorded ingredients imply compliance.
2. Sanitize Negation Phrases: Exclude phrases such as not suitable for vegans, non-vegan, may contain milk from triggering positive dietary flags.
3. Cross-Validate with Animal Tokens: Ensure products containing tokens from ANIMAL_TOKENS in black_box_audit.py cannot be tagged is_vegan: True.
4. Low Calorie Support: Add is_low_calorie based on Canadian regulations (<= 40 kcal / 100g).

Acceptance Criteria

• Rules are explicit and deterministic.
• Missing data is not incorrectly interpreted as a positive attribute (missing ingredients -> is_palm_oil_free: None).
• Negation phrases like not suitable for vegans do not evaluate to is_vegan: True.
• Products containing animal ingredients are never tagged as vegan or vegetarian.
• Existing tags are regression-tested.
• At least 15 new test cases added in backend/tests/test_nutrition.py.
• Existing behaviour is not regressed.
• Appropriate regression tests are added.
• Documentation is updated where necessary.

Implementation Prompt

First inspect the existing implementation and tests before making changes. Understand the current behaviour and identify the smallest appropriate change. Implement the requested functionality without unrelated refactoring. Add or update regression tests. Run the relevant tests, linting/type checks, and verification commands. Do not modify unrelated components.

Inspect `backend/builders/search_document_builder.py`, `backend/utils/off_parser.py`, and `backend/evaluation/black_box_audit.py`.
Refactor the dietary flag computation in SearchDocumentBuilder:
1. Fix the palm oil bug: if ingredients_text is empty or missing, is_palm_oil_free must not default to True.
2. Prevent negation false positives (e.g. "not suitable for vegans" or "non-vegan" must not produce is_vegan=True).
3. Check for animal ingredient tokens before granting is_vegan or is_vegetarian.
4. Support clean 3-state evaluation where missing data evaluates to None rather than false certainty.
5. Add comprehensive unit tests in `backend/tests/test_nutrition.py` covering clean cases and adversarial edge cases.
Do not use LLM inference for this issue.

Verification Plan

Run nutrition tests and audit verification:

pytest backend/tests/test_nutrition.py backend/tests/test_nutrition_ranking.py -v
python backend/evaluation/verify_nutrition.py

Dependencies

None

Maintainer Decision Required

No maintainer decision required.

Out of Scope

• Using an LLM during data ingestion or classification.
• Changing OpenSearch index mappings for unrelated fields.

Relevant Files

• backend/builders/search_document_builder.py
• backend/utils/off_parser.py
• backend/evaluation/black_box_audit.py
• backend/tests/test_nutrition.py
• backend/tests/test_nutrition_ranking.py