Initial commit for Field Mapping POC using LLM

lending-poc/modules/field_mapping_poc/README.md

@@ -0,0 +1,163 @@
+# LegalAI Field Mapping POC
+
+Generalized, schema-agnostic field mapping using a local Ollama model
+(default: `gemma3:4b`). Takes raw OCR/translated text + a target
+schema (arbitrary JSON, not a fixed Pydantic model) and returns a
+schema-shaped JSON with values filled in, `null` where nothing was
+found, and any extra fields the model discovers flagged separately.
+
+## Project layout
+
+```
+field_mapping_poc/
+├── config.py                    # model name, ollama host, retries, flagging keys
+├── main.py                      # CLI demo entry point
+├── core/
+│   ├── ollama_client.py         # thin wrapper around the ollama lib + retries
+│   ├── prompt_builder.py        # builds the system/user prompt from schema + text
+│   ├── response_parser.py       # JSON repair + schema reconciliation
+│   └── mapper.py                # FieldMapper — orchestrates the three above
+├── schemas/examples/
+│   ├── salary_slip.json         # flat schema example
+│   └── identity_card.json       # nested schema example (address object)
+├── samples/
+│   └── sample_ocr_text.txt      # messy OCR-style salary slip for the demo
+```
+
+## Setup
+
+```bash
+pip install -r requirements.txt --break-system-packages   # or use a venv
+
+# pull the model once
+ollama pull gemma3:4b
+ollama serve   # if not already running
+```
+
+## Run the demo
+
+```bash
+python main.py
+# or point it at a different schema/text pair
+python main.py --schema schemas/examples/identity_card.json --text samples/sample_ocr_text.txt
+```
+
+
+## HOW TO USE THIS IN THIS PROJECT
+
+To integrate this POC into the wider project pipeline, you only need to interact with the `FieldMapper` object.
+
+### 1. Object to create
+
+Create an instance of `FieldMapper` (located in `core/mapper.py`). By default, it will initialize its own `OllamaClient`, but you can also pass in a custom configured client if needed.
+
+```python
+from core.mapper import FieldMapper
+
+mapper = FieldMapper()
+```
+
+### 2. Method to call
+
+Call the `map_fields()` method on your `FieldMapper` instance.
+
+### 3. Parameters to give
+
+The `map_fields` method takes two arguments:
+- `schema` (`Dict[str, Any]`): An arbitrary JSON dictionary describing the expected fields (keys are field names, values are type/description hints).
+- `document_text` (`str`): The raw OCR (or translated) text that you want to extract information from.
+
+### Example Usage
+
+```python
+from core.mapper import FieldMapper
+
+# 1. Define your schema
+target_schema = {
+    "employeeName": "string",
+    "grossSalary": "number",
+    "dateOfJoining": "date (DD-MM-YYYY)"
+}
+
+# 2. Provide the raw text
+ocr_text = "..."
+
+# 3. Create the mapper and extract fields
+mapper = FieldMapper()
+extracted_data = mapper.map_fields(schema=target_schema, document_text=ocr_text)
+
+print(extracted_data)
+```
+
+## Design notes
+
+**Why the schema isn't a Pydantic model.** This needs to work across
+arbitrary document types (salary slips, identity cards, court orders,
+affidavits, ...) without writing a new Python class per type. So the
+"schema" is just a JSON object where each key is a field name and each
+value is a short type/description hint (`"string"`, `"number"`,
+`"date (DD-MM-YYYY)"`). It can nest (see `identity_card.json`'s
+`address` object). `FieldMapper.map_fields(schema, text)` works the
+same way regardless of what schema you pass in.
+
+**Extra-field convention.** The model is explicitly allowed to surface
+fields it's confident about that aren't in the schema (e.g. a UAN
+number on a salary slip). These stay at the same nesting level as
+everything else ("flat"), but get wrapped:
+
+```json
+"uanNumber": { "value": "101234567890", "source": "llm_added" }
+```
+
+while schema-defined fields stay as plain values:
+
+```json
+"employeeName": "Ishaan Deshmukh"
+```
+
+This means code that only cares about the schema fields doesn't need
+to change at all — it can keep reading plain values. Code that wants
+to review/promote extra fields (e.g. before Neo4j insertion) can just
+filter on `isinstance(v, dict) and v.get("source") == "llm_added"`.
+
+**Reliability layers.**
+1. `format="json"` is passed to Ollama, which constrains the model to
+   emit syntactically valid JSON (supported by Gemma 3 and other
+   JSON-mode-capable models).
+2. `response_parser.py` still does a repair pass (strips stray
+   markdown fences, extracts the `{...}` substring) in case the model
+   wraps the JSON in commentary anyway.
+3. `reconcile_with_schema()` guarantees every schema key exists in the
+   final output — even if the model silently dropped one — by
+   filling it with `null`.
+4. `ollama_client.py` retries transient failures with linear backoff
+   (configurable via `OLLAMA_MAX_RETRIES`).
+
+**What's deliberately out of scope for this POC** (per the ask):
+- Neo4j insertion — this only produces the clean per-document JSON
+  that would later be written to the graph.
+- Confidence scoring per field.
+- Batching / async processing across many documents at once.
+- The LoRA/QLoRA fine-tuning step — this POC targets a base/instruct
+  Gemma model via prompting only, to establish a baseline before any
+  fine-tuning work.
+
+## Suggested next steps
+
+- Run this against a handful of real document types (salary slip,
+  Aadhaar/PAN, property extract, etc.) and eyeball how often
+  `llm_added` fields are genuinely useful vs. noise — that ratio
+  should inform whether the "add new fields" permission stays as
+  loose as it is now, or gets tightened (e.g. require a `confidence`
+  field, or a controlled vocabulary of allowed extra fields per
+  document type).
+- Once field mapping is trusted, this is the natural point to bolt on
+  the Neo4j writer: schema fields become properties on a `:Document`
+  node (or field-specific nodes, depending on how the graph model is
+  set up for cross-document consistency checks), and `llm_added`
+  fields can be written with an extra `source: "llm_added"` property
+  so later Cypher queries can distinguish provenance.
+- If `gemma3:4b` under-performs on domain-specific field names (legal
+  terminology, regional document formats), that's the point to swap
+  in the QLoRA-fine-tuned checkpoint — `ollama_client.py` only needs
+  `OLLAMA_MODEL` changed, nothing else in the pipeline changes.

lending-poc/modules/field_mapping_poc/config.py

@@ -0,0 +1,37 @@
+"""
+Central configuration for the Field Mapping POC.
+
+Keep all environment-tunable values here so nothing is hardcoded deep
+inside business logic. Override any of these via environment variables
+without touching code.
+"""
+import os
+from dataclasses import dataclass
+
+
+@dataclass(frozen=True)
+class OllamaConfig:
+    host: str = os.getenv("OLLAMA_HOST", "http://localhost:11434")
+    model: str = os.getenv("OLLAMA_MODEL", "gemma3:4b")
+    temperature: float = float(os.getenv("OLLAMA_TEMPERATURE", "0.0"))
+    num_ctx: int = int(os.getenv("OLLAMA_NUM_CTX", "8192"))
+    request_timeout: int = int(os.getenv("OLLAMA_TIMEOUT_SECONDS", "120"))
+    max_retries: int = int(os.getenv("OLLAMA_MAX_RETRIES", "2"))
+
+
+OLLAMA = OllamaConfig()
+
+# --- Extra-field flagging convention -----------------------------------
+# Fields that ARE part of the caller's target schema are passed through
+# as plain values (no wrapper) so existing consumers of the schema shape
+# don't need to change.
+#
+# Fields the model discovers that are NOT part of the schema are kept
+# flat, at the same nesting level, but wrapped like:
+#   "uanNumber": {"value": "101234567890", "source": "llm_added"}
+# so downstream code (e.g. the future Neo4j writer) can easily filter
+# them out, route them through review, or promote them into the schema.
+EXTRA_FIELD_VALUE_KEY = "value"
+EXTRA_FIELD_SOURCE_KEY = "source"
+EXTRA_FIELD_SOURCE_TAG = "llm_added"
+SCHEMA_FIELD_SOURCE_TAG = "schema"

lending-poc/modules/field_mapping_poc/core/mapper.py

@@ -0,0 +1,54 @@
+"""
+FieldMapper: the main entry point of the POC.
+
+Given (target_schema, document_text) it returns a schema-shaped JSON
+with values filled in from the text, nulls where nothing was found,
+and any extra model-discovered fields flagged with a source marker.
+"""
+from __future__ import annotations
+
+import logging
+from typing import Any, Dict, Optional
+
+from core.ollama_client import OllamaClient
+from core.prompt_builder import build_system_prompt, build_user_prompt
+from core.response_parser import ResponseParseError, normalize_response
+
+logger = logging.getLogger(__name__)
+
+
+class FieldMapper:
+    def __init__(self, client: Optional[OllamaClient] = None):
+        self.client = client or OllamaClient()
+
+    def map_fields(
+        self,
+        schema: Dict[str, Any],
+        document_text: str,
+    ) -> Dict[str, Any]:
+        """
+        :param schema: arbitrary JSON describing expected fields
+                        (values are type/description hints, not data).
+        :param document_text: raw OCR (post-translation) text.
+        :return: schema-shaped dict, values filled or null, with any
+                 extra fields flagged as {"value": ..., "source": "llm_added"}.
+        :raises ValueError: on empty inputs.
+        :raises OllamaClientError: if the model backend fails.
+        :raises ResponseParseError: if the model's output can't be
+                                     salvaged into valid JSON.
+        """
+        if not document_text or not document_text.strip():
+            raise ValueError("document_text is empty")
+        if not schema:
+            raise ValueError("schema must not be empty")
+
+        system_prompt = build_system_prompt()
+        user_prompt = build_user_prompt(schema, document_text)
+
+        raw_response = self.client.generate_json(system_prompt, user_prompt)
+
+        try:
+            return normalize_response(schema, raw_response)
+        except ResponseParseError:
+            logger.error("Failed to parse model response:\n%s", raw_response)
+            raise

lending-poc/modules/field_mapping_poc/core/ollama_client.py

@@ -0,0 +1,78 @@
+"""
+Thin wrapper around the `ollama` python library.
+
+Keeping this isolated means:
+- if Ollama gets swapped for vLLM / a hosted endpoint later, this is
+  the only file that needs to change.
+- retry / timeout / error-handling logic lives in exactly one place.
+"""
+from __future__ import annotations
+
+import logging
+import time
+from typing import Optional
+
+import ollama
+
+from config import OLLAMA
+
+logger = logging.getLogger(__name__)
+
+
+class OllamaClientError(Exception):
+    """Raised when the Ollama backend fails after all retries."""
+
+
+class OllamaClient:
+    def __init__(
+        self,
+        model: str = OLLAMA.model,
+        host: str = OLLAMA.host,
+        temperature: float = OLLAMA.temperature,
+        num_ctx: int = OLLAMA.num_ctx,
+        max_retries: int = OLLAMA.max_retries,
+    ):
+        self.model = model
+        self.temperature = temperature
+        self.num_ctx = num_ctx
+        self.max_retries = max_retries
+        self._client = ollama.Client(host=host, timeout=OLLAMA.request_timeout)
+
+    def generate_json(self, system_prompt: str, user_prompt: str) -> str:
+        """
+        Calls the model in JSON mode and returns the raw string response.
+        Retries on transient failures with linear backoff (1s, 2s, ...).
+        """
+        last_error: Optional[Exception] = None
+
+        for attempt in range(1, self.max_retries + 2):
+            try:
+                response = self._client.chat(
+                    model=self.model,
+                    format="json",  # forces the model to emit valid JSON only
+                    options={
+                        "temperature": self.temperature,
+                        "num_ctx": self.num_ctx,
+                    },
+                    messages=[
+                        {"role": "system", "content": system_prompt},
+                        {"role": "user", "content": user_prompt},
+                    ],
+                )
+                content = response["message"]["content"]
+                if not content or not content.strip():
+                    raise OllamaClientError("Model returned empty content")
+                return content
+
+            except Exception as exc:  # noqa: BLE001 - retry on anything transient
+                last_error = exc
+                logger.warning(
+                    "Ollama call failed (attempt %d/%d): %s",
+                    attempt, self.max_retries + 1, exc,
+                )
+                if attempt <= self.max_retries:
+                    time.sleep(attempt)
+
+        raise OllamaClientError(
+            f"Ollama call failed after {self.max_retries + 1} attempts"
+        ) from last_error

lending-poc/modules/field_mapping_poc/core/prompt_builder.py

@@ -0,0 +1,58 @@
+"""
+Builds the system + user prompt sent to the model.
+
+The schema is intentionally NOT a strict Pydantic model — callers pass
+an arbitrary JSON object describing the fields they expect (a "target
+shape"), and the model returns a same-shaped JSON with values filled
+in, `null` where nothing was found, and permission to append genuinely
+new fields it discovers.
+"""
+from __future__ import annotations
+
+import json
+from typing import Any, Dict
+
+SYSTEM_PROMPT = """You are a precise information-extraction engine used in a \
+document processing pipeline. You will be given:
+1. A JSON "target schema" describing the fields expected in this document type. \
+Each key is a field name; each value is a short type/description hint \
+(e.g. "string", "number", "date (DD-MM-YYYY)"), not the actual value.
+2. Raw OCR-extracted text of a document (may contain OCR noise, spacing \
+errors, or be a translation of a non-English original).
+
+Your job:
+- Fill in the target schema with values found in the text.
+- If a field's value cannot be confidently found in the text, set it to null. \
+Never guess or fabricate a value.
+- Preserve the exact key names and nesting structure of the target schema.
+- Extract numbers as they appear in the text; do not silently reformat \
+currency separators unless it is clearly OCR noise.
+- If you notice other clearly-named, clearly-valued fields in the text \
+that are NOT part of the target schema but would be genuinely useful \
+for this document type (e.g. a UAN number on a salary slip), ADD them \
+to your JSON output at the same nesting level where they logically belong. \
+Only add fields you are confident about — do not pad the output with guesses.
+- Return ONLY a single valid JSON object. No commentary, no markdown fences, \
+no explanation text before or after the JSON.
+"""
+
+USER_PROMPT_TEMPLATE = """TARGET SCHEMA:
+{schema}
+
+DOCUMENT TEXT:
+\"\"\"
+{document_text}
+\"\"\"
+
+Return the filled JSON now."""
+
+
+def build_system_prompt() -> str:
+    return SYSTEM_PROMPT
+
+
+def build_user_prompt(schema: Dict[str, Any], document_text: str) -> str:
+    return USER_PROMPT_TEMPLATE.format(
+        schema=json.dumps(schema, indent=2, ensure_ascii=False),
+        document_text=document_text.strip(),
+    )