Website-Grounded RAG Agent

This project requires Python 3.11.

A production-quality, assessment-scoped RAG system that crawls a configured public website, embeds content with Mistral AI, stores vectors in local FAISS, and answers questions grounded only in retrieved website content — with source URLs, refusal behavior, token/cost tracking, evaluation, CLI, FastAPI, and Docker.

AI provider: Mistral AI only. This project intentionally does not use OpenAI.

Assessment Overview

Hiring assessment demonstrating:

Website crawling with domain/SSRF safety
Clean content extraction and chunking
Mistral embeddings + chat
Local FAISS retrieval
Strict grounding / hallucination controls
Citations, evaluation, cost analysis, tests, Docker

Problem Statement

Build an agent that can answer questions about a specific website using only that site's content. If the site does not support an answer (or the question is misleading/unrelated), the system must refuse rather than invent.

Key Requirements

Requirement	Implementation
Python 3.11	Enforced in docs, `pyproject.toml`, Docker base image
Mistral LLM + embeddings	Official `mistralai` SDK
No OpenAI	Not in dependencies or code
FAISS	Local `IndexFlatIP` (cosine via L2-normalization)
Grounding	System prompt + deterministic refusal
Sources	From retrieved chunk metadata only
Evaluation	10 questions, deterministic checks
FastAPI + CLI + Docker	Included

Verified Final State

These results were produced by a real crawl, ingestion, and evaluation run — not fabricated.

Metric	Value
Chat model	`ministral-3b-latest`
Embedding model	`mistral-embed`
Embedding dimensions	1024
Pages crawled	20
Chunks created	78
FAISS vectors	78
Unit tests	34 passed, 0 failed
Evaluation questions	10
Pass rate	10/10 (100%)
Correct refusal rate	100%
Source citation rate	100%
Retrieval success rate	100%
Avg. evaluation latency	~2,700 ms
Total evaluation cost	$0.0032542
Avg. cost per query	$0.00032542

Features

Domain-restricted crawler with SSRF protection (no localhost / private IPs / file://)
Sitemap + robots.txt aware crawling
Trafilatura-first content extraction with BeautifulSoup fallback
Recursive character text splitting (chunk size 1000, overlap 150)
Mistral embeddings (mistral-embed, 1024 dimensions)
FAISS IndexFlatIP with L2-normalized vectors (cosine similarity)
Idempotent ingestion with content-version hashing
Deterministic refusal when retrieval score is below threshold (no LLM call)
Structured [Source N] context blocks; citations from metadata only
Token usage tracking from actual Mistral API response fields
Per-query cost estimation (requires configured pricing env vars)
Latency tracking per query
FastAPI with /health, /stats, /query, /admin/reload-index
CLI scripts for every pipeline stage
34 unit tests (no API key required)
Docker support (python:3.11-slim)

Architecture / Pipeline

See docs/architecture.md for Mermaid diagrams and component detail.

Ingestion pipeline:

Website → Crawl → Clean/Extract → Chunk → Mistral Embeddings → FAISS

Query pipeline:

Question → Validate → Embed → FAISS Similarity Search → Threshold Check
    → [Refuse if below threshold]
    → Format [Source N] context blocks
    → Mistral LLM (ministral-3b-latest)
    → Answer + Sources + Usage + Cost + Latency

Query Flow (step by step)

Validate question (non-empty, max 2000 chars)
Embed with mistral-embed
Retrieve top-5 chunks from FAISS (RETRIEVAL_K=5)
If empty or all scores below SIMILARITY_THRESHOLD → refuse without calling the LLM
Format structured context blocks [Source N]
Call ministral-3b-latest at temperature 0.1
Map citations from retrieved chunk metadata (never LLM-invented URLs)
Return answer, grounded flag, sources, usage, cost, latency

Tech Stack

Python 3.11
FastAPI + Uvicorn + Pydantic / pydantic-settings
LangChain RecursiveCharacterTextSplitter
Mistral AI (mistralai) — chat + embeddings
FAISS (faiss-cpu) — IndexFlatIP
httpx + BeautifulSoup4 + Trafilatura
pytest, Ruff
Docker (python:3.11-slim)

LangGraph was not added — a single clear RAG chain is sufficient and easier to explain for this assessment.

Why Mistral AI

Single provider for both generation and embeddings
Official Python SDK with usage metadata
Avoids OpenAI dependency for this assessment
Configurable model IDs via environment (no obsolete hardcoding)

Why FAISS

Fast local similarity search
No external DB/ops for assessment scope
Easy to persist (index.faiss + metadata + manifest)
Deterministic reload for demos and tests

Similarity: vectors are L2-normalized; IndexFlatIP ≈ cosine similarity.

Why LangChain

Mature RecursiveCharacterTextSplitter
Familiar RAG building blocks for interviews
Kept thin — core providers use the official Mistral SDK for reliable usage tracking

Why Python 3.11

Explicit assessment requirement
Stable typing / performance sweet spot
Docker and requires-python = "==3.11.*" pin the same version

Do not run this project on Python 3.10 or 3.13.

Project Structure

app/                  # Application package
  api/                # FastAPI routes + schemas
  crawler/            # Crawl, sitemap, robots, URL safety
  ingestion/          # Load, clean, chunk, pipeline
  embeddings/         # EmbeddingProvider + Mistral
  llm/                # LLMProvider + Mistral
  vectorstore/        # FAISS + manifest
  rag/                # Prompts, grounding, retriever, chain
  evaluation/         # Evaluator, metrics, cost
  config/             # Settings
  models/             # Domain models
  logging/            # Secret-redacting logger
  main.py             # FastAPI app
scripts/              # crawl, ingest, query, evaluate, validate_config
tests/                # unit + optional integration
data/                 # raw / processed / manifests
vector_store/         # FAISS artifacts
evaluation/           # questions.json + results.json
docs/architecture.md

Prerequisites

Windows (documented) or any OS with Python 3.11
Mistral API key: https://console.mistral.ai/
Network access to the target website

Python 3.11 Setup

Verify the launcher can see 3.11:

py -3.11 --version

Expected: Python 3.11.x

If py is unavailable, use the full path to the 3.11 installer binary, for example:

& "C:\Users\<YOU>\AppData\Local\Programs\Python\Python311\python.exe" --version

Virtual Environment

Always create the venv with Python 3.11 (do not use whatever python is first on PATH — it may be 3.13):

cd "C:\Users\HP\Desktop\Projects\Website Grounded RAG Agent"
py -3.11 -m venv .venv
.venv\Scripts\activate
python --version

python --version must report Python 3.11.x.

Configuration / Environment Variables

Copy .env.example → .env
Fill in required values (see table below)
Never commit .env

Variable	Required	Default	Description
`MISTRAL_API_KEY`	Yes	—	From https://console.mistral.ai/
`MISTRAL_MODEL`	Yes	—	Chat model ID (e.g. `ministral-3b-latest`)
`MISTRAL_EMBEDDING_MODEL`	No	`mistral-embed`	Embedding model ID
`MISTRAL_TEMPERATURE`	No	`0.1`	LLM temperature
`MISTRAL_MAX_OUTPUT_TOKENS`	No	`800`	Max tokens in LLM response
`MISTRAL_LLM_INPUT_PRICE_PER_1M`	No	—	USD per 1M input tokens
`MISTRAL_LLM_OUTPUT_PRICE_PER_1M`	No	—	USD per 1M output tokens
`MISTRAL_EMBEDDING_PRICE_PER_1M`	No	—	USD per 1M embedding tokens
`WEBSITE_BASE_URL`	No	`https://myadvice.com/`	Target website to crawl
`MAX_PAGES`	No	`20`	Max pages to crawl
`CRAWL_DELAY_SECONDS`	No	`1.0`	Delay between requests
`CHUNK_SIZE`	No	`1000`	Characters per chunk
`CHUNK_OVERLAP`	No	`150`	Overlap between chunks
`RETRIEVAL_K`	No	`5`	Top-k chunks to retrieve
`SIMILARITY_THRESHOLD`	No	`0.35`	Min cosine score; below → refuse
`ENABLE_QUERY_REWRITE`	No	`false`	Query rewriting (not enabled by default)

If pricing variables are unset, the system reports Pricing not configured instead of inventing numbers.

Installation

.venv\Scripts\activate
python -m pip install --upgrade pip
pip install -r requirements.txt

Confirm no OpenAI package:

pip show openai
# should report that the package is not found

Running the Project

Full pipeline from scratch:

python scripts/validate_config.py       # verify env + optional model check
python scripts/crawl.py                 # crawl website → data/raw/
python scripts/ingest.py                # embed + index → vector_store/
python scripts/query.py "Your question" # single query
python scripts/evaluate.py              # run evaluation suite

CLI Usage

Validate config

python scripts/validate_config.py
python scripts/validate_config.py --verify-model   # optional live model check

Crawl

python scripts/crawl.py

Writes data/raw/pages.jsonl and data/raw/crawl_stats.json. Default MAX_PAGES=20.

Ingest

python scripts/ingest.py
python scripts/ingest.py --force   # force rebuild even if content hash matches

Idempotent: unchanged content hash + matching config → skips re-embedding.

Query

python scripts/query.py "What is MyAdvice?"
python scripts/query.py   # interactive prompt

Output includes: answer, grounded flag, source URLs, retrieval scores, token usage, cost projections (1 / 100 / 1k / 10k queries), latency.

Evaluate

python scripts/evaluate.py

Writes evaluation/results.json and prints a console summary.

Vector Store

Persisted under vector_store/:

File	Purpose
`index.faiss`	FAISS index (78 vectors, 1024 dimensions)
`metadata.json`	Chunks + source metadata
`manifest.json`	Model, dimension, chunk config, website, content hash

Incompatible manifests refuse to load and tell you to rebuild.

FastAPI API Usage

uvicorn app.main:app --reload --host 0.0.0.0 --port 8000

Method	Path	Description
GET	`/health`	Liveness + config flags
POST	`/query`	Grounded Q&A
GET	`/stats`	Index stats (vector count, dimensions, source website)
POST	`/admin/reload-index`	Reload FAISS from disk (no secrets exposed)

Example query:

curl -X POST http://127.0.0.1:8000/query `
  -H "Content-Type: application/json" `
  -d "{\"question\":\"What is MyAdvice?\"}"

Example response shape:

{
  "answer": "...",
  "grounded": true,
  "sources": [{"url": "...", "title": "...", "score": 0.85}],
  "usage": {"input_tokens": 1068, "output_tokens": 184, "total_tokens": 1252},
  "cost": {"total_estimated_cost_usd": 0.000271, "pricing_configured": true},
  "latency_ms": 1835.65
}

Evaluation

Dataset: evaluation/questions.json — 10 questions across 5 categories:

Category	Count	Description
straightforward	2	Direct factual questions about the site
paraphrased	2	Same intent, different wording
multi-page	2	Require synthesizing multiple pages
misleading	2	False premises that must be refused
unanswerable	2	Topics not covered by the site

Evaluation Results (measured)

Metric	Result
Total questions	10
Passed	10 (100%)
Correct refusal rate	100%
Source citation rate	100%
Retrieval success rate	100%
Average latency	~2,700 ms
Total cost	$0.0032542
Average cost/query	$0.00032542

Results are written to evaluation/results.json after each run.

Evaluation methodology: Pass/fail uses deterministic checks — refusal detection, grounded flag, required keyword presence, and expected source URL presence. This is not full semantic equivalence scoring or LLM-as-judge evaluation. Semantic correctness requires human review.

Testing

Unit tests mock Mistral — no API key required:

pytest tests/unit -q

Result: 34 passed, 0 failed.

Test files cover: API routes, chunking, cost calculation, embeddings, content extraction, FAISS store, grounding logic, sitemap parsing, URL utilities.

Optional live integration tests (requires API key):

pytest tests/integration -m integration -q

Cost Analysis

Pricing is configuration-driven via environment variables:

MISTRAL_LLM_INPUT_PRICE_PER_1M=
MISTRAL_LLM_OUTPUT_PRICE_PER_1M=
MISTRAL_EMBEDDING_PRICE_PER_1M=

If all three are set, the system computes tokens / 1,000,000 * price_usd per query.
If any are unset, the system reports Pricing not configured — it never invents numbers.
Token counts use actual Mistral API usage fields when present.
The query CLI prints cost projections at 1 / 100 / 1k / 10k query volumes.

The evaluation run above used configured pricing and produced a total cost of $0.0032542 across 10 queries.

Grounding & Refusal Behavior

Grounding strategy

Strict system prompt: answer only from provided context, no outside knowledge
Structured [Source N] context blocks passed to the LLM
Citations projected from retrieved chunk metadata — never from LLM output
Low temperature (0.1) to reduce hallucination

Refusal behavior

If retrieval returns no chunks or all scores are below SIMILARITY_THRESHOLD (default 0.35), the system refuses without calling the LLM
If the LLM cannot answer from context, it returns a standard refusal string
Misleading premises (false claims about the site) are refused
Unrelated questions (e.g., science facts) are refused

Hallucination prevention

Context-only authority enforced in system prompt
LLM skipped entirely on empty/low-similarity retrieval
Deterministic refusal string for insufficient evidence
Source URLs come only from retrieved chunk metadata
Temperature fixed at 0.1

Source traceability

Every chunk stores: source_url, page_title, source_domain, chunk_id, document_id, content_hash. Final citations are projected from retrieval hits — the LLM cannot invent URLs.

Security

MISTRAL_API_KEY from environment only
.env gitignored
Secret-redacting logger
Domain-restricted crawler; blocks localhost / private IPs / file://
Query API does not accept crawl URLs
Admin reload does not expose secrets

Content Extraction Decision

Trafilatura is preferred for main-article extraction (less chrome/nav).
BeautifulSoup is the fallback for structure-aware extraction when Trafilatura returns little. Both paths strip scripts/styles; the pipeline never embeds raw HTML.

Docker

Base image: python:3.11-slim (not 3.10 / 3.13).

docker build -t website-grounded-rag .
docker compose up --build

Mounts data/, vector_store/, and evaluation/. Ensure you have crawled + ingested before querying the API container.

Project Limitations

Single-website scope; local FAISS is not multi-tenant or production-scale
Evaluation uses deterministic checks only — not full semantic scoring or LLM-as-judge
Crawl quality depends on site structure; JS-rendered content may not be captured
Pricing estimates are only as accurate as the configured rate values
Query rewriting is not enabled by default (ENABLE_QUERY_REWRITE=false)
No authentication on admin endpoints (assessment scope)

Future Improvements

Hybrid BM25 + dense retrieval
Incremental recrawl / change detection webhooks
Optional LangGraph multi-step agents for complex research
Stronger eval with annotated gold answers and LLM-as-judge scoring
AuthN for admin endpoints in production
Managed vector DB (e.g., OpenSearch, Pinecone) for production scale

Error Handling

Actionable errors for: missing API key/model, incompatible FAISS index, network failures, empty crawls, provider errors, and validation failures. Secrets are never logged.

Interview Discussion Points

Why Mistral? Single-provider LLM+embeddings; assessment forbids OpenAI.
Why mistral-embed? Native Mistral embedding model; dimension taken from API output, not hard-coded.
Why ministral-3b-latest? Fast, cost-efficient chat model available in the Mistral account used for this assessment.
Why FAISS? Local, fast, reproducible, zero infra.
Why LangChain? Splitter + clear RAG glue without over-frameworking.
Why Python 3.11? Explicit requirement; avoid 3.10/3.13 drift.
Chunking? Recursive splitter; size 1000 / overlap 150 balances context vs. specificity.
Why overlap? Preserve sentence continuity across chunk boundaries.
Why top-k=5? Enough multi-page evidence without drowning the prompt.
Hallucination prevention? Prompt + threshold refusal + metadata-only citations.
Unanswerable? Deterministic refusal path before LLM call + LLM refusal detection.
Source URLs? Always from retrieved chunk metadata, never from LLM output.
Multi-page? Multiple chunks, Source IDs grouped by URL.
Costs? tokens / 1e6 * configured USD; else "Pricing not configured".
Content changes? Content-version hash; --force rebuild.
Limitations? See above.
Scale to production? Managed vector DB, auth, queue, monitoring, eval harness in CI.
Why no external vector DB / frontend? Assessment scope and explainability.

License

This project is licensed under the MIT License.

Commands Cheat Sheet

py -3.11 -m venv .venv
.venv\Scripts\activate
python --version
pip install -r requirements.txt
python scripts/validate_config.py
python scripts/crawl.py
python scripts/ingest.py
python scripts/query.py "Your question"
python scripts/evaluate.py
pytest tests/unit -q
uvicorn app.main:app --reload --port 8000

Name		Name	Last commit message	Last commit date
app		app
data		data
docs		docs
evaluation		evaluation
scripts		scripts
tests		tests
vector_store		vector_store
.env.example		.env.example
.gitignore		.gitignore
Dockerfile		Dockerfile
LICENSE		LICENSE
README.md		README.md
docker-compose.yml		docker-compose.yml
pyproject.toml		pyproject.toml
requirements.txt		requirements.txt

Latest commit

History

Folders and files

Repository files navigation