Product data completeness scoring

[SaitejaKommi]

Priority

P1

Area

Data Modeling / Ranking

Problem

Ingestion reads an opaque completeness column from the raw Parquet export. This upstream number is an undocumented black box, often missing or inconsistent. Because completeness * 0.15 is applied as an OpenSearch function score boost on every search, opaque completeness directly impacts search ranking without local explainability.

Why This Matters

Search ranking should be explainable, repeatable, and transparent. If a high-quality product is assigned an arbitrary low completeness value upstream, its BM25 ranking is unfairly penalized. A local deterministic scoring formula ensures consistent ranking behavior across all products.

Current Behaviour

• backend/adapters/off_adapter.py reads {comp_col} as completeness from raw Parquet.
• backend/builders/search_document_builder.py assigns metadata["completeness"] = raw.completeness or defaults to 0.0.
• backend/repositories/opensearch_repository.py injects field_value_factor on metadata.completeness with weight 0.15.
• No documented formula exists in the codebase explaining what makes a product complete.

Expected Behaviour

• Transparent, deterministic completeness score (0.0 to 1.0) computed locally from Golden Product Record fields:
  • Product name (+0.20)
  • Brand (+0.10)
  • Categories (+0.10)
  • Ingredients (+0.15)
  • Core nutrients (+0.25)
  • Product image URL (+0.10)
  • Nutri-Score / NOVA (+0.10)
• Missing information must NOT automatically mean that an attribute is absent.
• Full breakdown stored in metadata for explainability.
• Missing values handled gracefully without negative numbers or NaNs.

Proposed Implementation

1. Transparent Weighted Formula: Compute completeness dynamically in backend/builders/search_document_builder.py based on verified attributes totaling 1.0.
2. Score Breakdown: Store both completeness (float 0.0-1.0) and completeness_breakdown dictionary in doc.metadata.
3. Preserve Upstream Fallback: Document whether the local formula overrides or supplements the raw column.
4. Ranking Impact Evaluation: Benchmark search ranking impact before enabling the new score in production function scoring.

Acceptance Criteria

• Completeness dimensions and additive weights are clearly defined and sum to 1.0.
• Score calculation is deterministic and handles missing values without NaNs.
• Complete products score >= 0.85; minimal records score appropriately lower.
• Score breakdown dictionary is inspectable in document metadata.
• Potential ranking impact is evaluated with evaluate.py.
• Existing behaviour is not regressed.
• Appropriate regression tests are added.
• Documentation is updated where necessary.

Implementation Prompt

First inspect the existing implementation and tests before making changes. Understand the current behaviour and identify the smallest appropriate change. Implement the requested functionality without unrelated refactoring. Add or update regression tests. Run the relevant tests, linting/type checks, and verification commands. Do not modify unrelated components.

Inspect `backend/builders/search_document_builder.py`, `backend/adapters/off_adapter.py`, and `backend/retrieval/ranking.py`.
Implement a transparent, deterministic completeness scoring function in SearchDocumentBuilder:
1. Define clear additive weights for product_name, brand, category, ingredients, 5 core nutrients, image URL, and Nutri-Score totaling 1.0.
2. Compute the score dynamically during document building and attach both 'completeness' and a breakdown dictionary to metadata.
3. Ensure no NaN or negative values can be produced.
4. Add unit tests in `backend/tests/test_nutrition_ranking.py` validating completeness scoring across sample products.

Verification Plan

Run ranking and completeness tests:

pytest backend/tests/test_nutrition_ranking.py -v
curl "http://127.0.0.1:8000/product/0068100084124" | grep -o '"completeness":[^,]*'

Dependencies

None

Maintainer Decision Required

No maintainer decision required.

Out of Scope

• Penalizing products based on whether their ingredients are perceived as healthy.
• Making completeness the primary sort order ahead of lexical BM25 match quality.

Relevant Files

• backend/builders/search_document_builder.py
• backend/adapters/off_adapter.py
• backend/retrieval/ranking.py
• backend/tests/test_nutrition_ranking.py