cross-document validation Module - #11
Open
Ankita-Advitot wants to merge 1 commit into
Open
Conversation
There was a problem hiding this comment.
Pull request overview
Adds an end-to-end “document validation & decision” pipeline to lending-poc, including in-memory validation logic, DB persistence (SQLAlchemy + Alembic), and a new POST /cases API entrypoint to run the pipeline and store results.
Changes:
- Introduces the validation pipeline (golden record build, identity validation, business validation, scoring, decision) plus demo/edge-case scripts and workflow documentation.
- Adds Postgres persistence layer: SQLAlchemy models for cases/documents/golden_records/validation_results/pipeline_results and an Alembic migration (including
pgvector). - Adds encryption-at-rest for Aadhaar/PAN fields and replaces address “stub” embeddings with local
BAAI/bge-small-en-v1.5embeddings.
Reviewed changes
Copilot reviewed 33 out of 35 changed files in this pull request and generated 9 comments.
Show a summary per file
| File | Description |
|---|---|
| lending-poc/scripts/sample_case.json | Adds a realistic sample payload used for demos and manual validation. |
| lending-poc/scripts/run_demo.py | Adds a CLI demo runner for the in-memory pipeline. |
| lending-poc/scripts/edge_case_scenarios.py | Adds scenario-based pipeline runs to probe edge behaviors. |
| lending-poc/pyproject.toml | Adds dependencies for pgvector, cryptography, sentence-transformers, rapidfuzz. |
| lending-poc/docs/Workflow.md | Documents the request shape, outputs, and pipeline stages/logic. |
| lending-poc/docker-compose.yml | Switches DB image to pgvector-enabled Postgres and changes host port mapping. |
| lending-poc/app/services/validation_config.py | Centralizes thresholds/weights/constants for validation and decisioning. |
| lending-poc/app/services/scoring.py | Implements weighted score aggregation across validation results. |
| lending-poc/app/services/pipeline.py | Orchestrates golden record → validations → scoring → decision with audit log. |
| lending-poc/app/services/persistence.py | Persists a pipeline run and resolves doc_id→FK mapping + JSON-safe evidence. |
| lending-poc/app/services/identity_validation.py | Implements identity checks against the golden record + mandatory presence checks. |
| lending-poc/app/services/golden_record.py | Builds the golden record and computes address embeddings. |
| lending-poc/app/services/dto.py | Adds dataclasses/enums used across parsing, validation, scoring, decisioning. |
| lending-poc/app/services/decision_engine.py | Applies mandatory-field rules + score thresholds to produce PASS/FAIL/REVIEW. |
| lending-poc/app/services/case_parsing.py | Parses request JSON into DTOs for pipeline execution. |
| lending-poc/app/services/business_validation.py | Matches salary slips to bank transactions and computes employer/count checks. |
| lending-poc/app/services/init.py | Package marker for services. |
| lending-poc/app/schemas/case.py | Adds request/response models for POST /cases. |
| lending-poc/app/models/validation_result.py | Adds ORM model for persisted validation results. |
| lending-poc/app/models/types.py | Adds EncryptedString type decorator for encrypted-at-rest strings. |
| lending-poc/app/models/pipeline_result.py | Adds ORM model for persisted pipeline runs and review metadata fields. |
| lending-poc/app/models/golden_record.py | Adds ORM model for golden records including vector embeddings and encrypted fields. |
| lending-poc/app/models/document.py | Adds ORM model for stored document JSON + source reference. |
| lending-poc/app/models/case.py | Adds ORM model for cases and relationships. |
| lending-poc/app/models/init.py | Exposes ORM models for Alembic metadata registration. |
| lending-poc/app/matching/fuzzy.py | Implements fuzzy name/employer similarity (RapidFuzz + initials logic). |
| lending-poc/app/matching/exact.py | Implements exact/tri-state matching for Aadhaar, PAN, DOB. |
| lending-poc/app/matching/embeddings.py | Implements local address embeddings + cosine similarity. |
| lending-poc/app/matching/init.py | Package marker for matching. |
| lending-poc/app/main.py | Registers the new cases router. |
| lending-poc/app/config.py | Adds encryption key config setting. |
| lending-poc/app/api/cases.py | Adds POST /cases endpoint to run pipeline and persist results. |
| lending-poc/alembic/versions/12522c432f16_add_cases_documents_golden_records_.py | Creates tables/enums and enables the vector extension. |
| lending-poc/alembic/env.py | Registers models to support Alembic autogenerate. |
| lending-poc/.gitignore | Ignores local virtualenv directory. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
+13
to
+37
| @router.post("/cases", response_model=CaseCreateResponse) | ||
| async def create_case( | ||
| request: CaseCreateRequest, db: AsyncSession = Depends(get_db) | ||
| ) -> CaseCreateResponse: | ||
| case_input = parse_case(request.model_dump()) | ||
| pipeline_result = run_pipeline(case_input) | ||
| case = await save_pipeline_result(db, case_input, pipeline_result) | ||
|
|
||
| return CaseCreateResponse( | ||
| case_id=str(case.id), | ||
| applicant_ref=case_input.applicant_ref, | ||
| decision=pipeline_result.decision_result.decision.value, | ||
| overall_score=pipeline_result.decision_result.overall_score, | ||
| reasons=pipeline_result.decision_result.reasons, | ||
| validation_results=[ | ||
| ValidationResultOut( | ||
| check_type=r.check_type.value, | ||
| passed=r.passed, | ||
| score=r.score, | ||
| document_id=r.document_id, | ||
| evidence=r.evidence, | ||
| ) | ||
| for r in pipeline_result.validation_results | ||
| ], | ||
| ) |
Comment on lines
+105
to
+114
| if slip.salary_month is None: | ||
| return ValidationResult( | ||
| check_type=CheckType.SALARY_DATE, | ||
| passed=False, | ||
| score=0.0, | ||
| document_id=slip.doc_id, | ||
| failure_reason="missing_salary_month", | ||
| ) | ||
|
|
||
| window = _month_window(slip.salary_month) |
Comment on lines
+108
to
+119
| extracted_fields={ | ||
| "account_holder": case.bank_statement.name, | ||
| "transactions": [ | ||
| { | ||
| "narration": txn.narration, | ||
| "amount": txn.amount, | ||
| "date": txn.txn_date.isoformat() if txn.txn_date else None, | ||
| } | ||
| for txn in case.bank_statement.transactions | ||
| ], | ||
| }, | ||
| ), |
Comment on lines
+15
to
+26
| impl = String | ||
| cache_ok = True | ||
|
|
||
| def process_bind_param(self, value: str | None, dialect) -> str | None: | ||
| if value is None: | ||
| return None | ||
| return Fernet(settings.ENCRYPTION_KEY).encrypt(value.encode("utf-8")).decode("utf-8") | ||
|
|
||
| def process_result_value(self, value: str | None, dialect) -> str | None: | ||
| if value is None: | ||
| return None | ||
| return Fernet(settings.ENCRYPTION_KEY).decrypt(value.encode("utf-8")).decode("utf-8") |
Comment on lines
+93
to
+98
| extracted_fields={ | ||
| "employee_name": slip.name, | ||
| "employer": slip.employer_name, | ||
| "net_salary": slip.net_salary, | ||
| "salary_month": slip.salary_month.isoformat() if slip.salary_month else None, | ||
| }, |
|
|
||
| sys.path.insert(0, str(Path(__file__).resolve().parent.parent)) | ||
|
|
||
| from scripts.run_demo import parse_case # noqa: E402 |
Comment on lines
8
to
10
| ports: | ||
| - "5432:5432" | ||
| - "55432:5432" | ||
| volumes: |
Comment on lines
+7
to
+16
| from typing import Any | ||
|
|
||
| from pydantic import BaseModel | ||
|
|
||
|
|
||
| class DocumentIn(BaseModel): | ||
| doc_type: str | ||
| extracted_fields: dict[str, Any] | None = None | ||
| source_file_ref: str | None = None | ||
| salary_slips: list[dict[str, Any]] | None = None # only present when doc_type == SALARY_SLIP |
| DEBUG: bool = False | ||
| DATABASE_URL: str = "postgresql+asyncpg://postgres:postgres@localhost:5432/lending_poc" | ||
| LOG_LEVEL: str = "INFO" | ||
| ENCRYPTION_KEY: str = "" |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Title: Add DB schema, ORM models, and persistence wiring for the validation pipeline
Summary:
Adds SQLAlchemy models + Alembic migration for the 5-table schema (cases, documents, golden_records, validation_results, pipeline_results), including pgvector for address embeddings and app-level encryption for Aadhaar/PAN.
Wires the existing in-memory validation pipeline to Postgres via a new POST /cases endpoint — request comes in, pipeline runs, results are persisted in one transaction.
Switches address embeddings from a deterministic hash stub to real BAAI/bge-small-en-v1.5 embeddings (local, 384-dim).