FastAPI · Next.js · SQLite · Groq (Llama 3) · Plotly · scikit-learn · XGBoost
An AI-powered analytics assistant that ingests messy real-world datasets, automatically analyzes data quality, generates its own Python analysis code on the fly, runs machine learning model comparisons, and produces polished PDF reports — all from natural language questions.
- Screenshots
- How It Works
- Architecture
- Quick Start (Windows)
- Project Structure
- API Endpoints
- What You Can Ask the Chat
- Data Quality & Ingestion
- Engineering Challenges Solved
- Environment Variables
- Tech Stack
- License
|
Statistical summary
Full descriptive statistics generated from a natural language request
|
Correlation analysis
Auto-generated Pearson correlation heatmap
|
|
Category breakdown
Pie chart generated directly from a plain-English request
|
Anomaly detection
Isolation Forest with PCA projection, score distribution, and feature comparison
|
|
Regression comparison
Model comparison with feature importance, rendered from a single chat request
|
ML model comparison
Cross-validated Ridge/Random Forest/XGBoost comparison with a downloadable, self-contained model
|
Unlike a keyword-routed chatbot, CortexIQ's chat is a code-generation and execution pipeline:
- You ask a question in plain English ("show me a histogram of age", "predict charges", "cluster this data").
- Groq (Llama 3) generates real pandas/Plotly/scikit-learn Python code, using the dataset's exact column names and dtypes as context.
- The code runs in a sandboxed executor (restricted builtins, no imports) against your actual dataframe.
- If the code errors, CortexIQ automatically sends the error back to the LLM for up to 2 self-correction attempts before giving up.
- Results — a chart, a table, or a full ML model comparison — stream back to the browser over Server-Sent Events.
flowchart TD
subgraph Client["Frontend"]
FE[Next.js UI]
end
subgraph Backend["FastAPI Backend"]
API[API Routes]
ING["Ingestion Layer<br/>delimiter/encoding recovery,<br/>sentinel-value detection"]
EXEC[Sandboxed Code Executor]
MLENG["ML Engine<br/>anomaly · clustering · model comparison"]
RPT["PDF Report Generator<br/>ReportLab + Kaleido"]
end
GROQ["Groq LLM<br/>(Llama 3)"]
DB[(SQLite)]
FE -- "1. Upload dataset" --> API
API --> ING
ING --> DB
FE -- "2. Ask a question (SSE)" --> API
API -- "3. Generate code" --> GROQ
GROQ -- "Python code" --> EXEC
EXEC -- "4. Error → retry with feedback" --> GROQ
EXEC -- "5. Success" --> MLENG
EXEC --> RPT
MLENG -- "chart / table / downloadable model" --> API
RPT -- "branded PDF" --> API
API -- "Stream results" --> FE
- Python 3.10+
- Node.js 18+
- A free Groq API key
REM Run from the CortexIQ/ root folder
setup.batOr manually:
REM Terminal 1 — Backend
cd backend
python -m venv venv
venv\Scripts\activate
pip install -r requirements.txt
copy .env.example .env
REM Edit .env and add your GROQ_API_KEY
uvicorn app.main:app --reload --port 8000REM Terminal 2 — Frontend
cd frontend
npm install
npm run devOpen http://localhost:3000 in your browser.
CortexIQ/
├── backend/
│ ├── app/
│ │ ├── main.py # FastAPI app entry point
│ │ ├── core/
│ │ │ ├── config.py # Settings (pydantic-settings)
│ │ │ └── security.py # JWT auth utilities
│ │ ├── db/
│ │ │ └── database.py # SQLAlchemy engine + session
│ │ ├── models/ # SQLAlchemy ORM models
│ │ │ ├── user.py
│ │ │ ├── project.py
│ │ │ ├── dataset.py
│ │ │ └── conversation.py
│ │ ├── schemas/ # Pydantic request/response schemas
│ │ │ ├── auth.py
│ │ │ └── project.py
│ │ ├── api/routes/ # FastAPI routers
│ │ │ ├── auth.py # /auth/register, /auth/login, /auth/me
│ │ │ ├── projects.py # /projects CRUD
│ │ │ ├── datasets.py # upload, quality, preview, PDF report
│ │ │ ├── chat.py # /conversations + SSE code-execution chat
│ │ │ └── ml.py # dedicated ML task endpoints
│ │ └── services/
│ │ ├── ingestion.py # Robust parsing: encoding detection,
│ │ │ # delimiter/decimal recovery, sentinel-
│ │ │ # value (e.g. -200) detection
│ │ ├── analytics.py # Descriptive stats, correlation, charts
│ │ ├── code_executor.py # Sandboxed execution of LLM-generated code
│ │ ├── ml_engine.py # Anomaly detection, clustering, and
│ │ │ # classifier/regressor comparison —
│ │ │ # saves a self-contained, downloadable
│ │ │ # sklearn Pipeline (preprocessing baked in)
│ │ ├── report_generator.py # One-click branded PDF report (ReportLab
│ │ │ # + Plotly/Kaleido), with auto-generated
│ │ │ # insights and outlier/correlation charts
│ │ └── llm.py # Groq LLM service (code-gen + chat)
│ ├── requirements.txt
│ └── .env.example
│
├── frontend/
│ └── src/
│ ├── app/
│ │ ├── login/page.tsx # Auth page
│ │ ├── dashboard/
│ │ │ ├── layout.tsx # Auth guard + sidebar
│ │ │ ├── page.tsx # Projects list
│ │ │ └── projects/[projectId]/page.tsx # Project workspace
│ ├── components/
│ │ ├── dashboard/
│ │ │ ├── Sidebar.tsx
│ │ │ ├── DatasetUploader.tsx # Drag-and-drop upload
│ │ │ └── DatasetCard.tsx # Dataset status card
│ │ └── chat/
│ │ ├── ChatWindow.tsx # Full chat UI, SSE streaming,
│ │ │ # report + model download buttons
│ │ └── ChatMessage.tsx # Message bubble + Plotly chart +
│ │ # downloadable model button
│ ├── lib/
│ │ ├── api.ts # Axios API client + SSE streaming
│ │ └── store.ts # Zustand auth store
│ └── types/index.ts
│
├── setup.bat # Windows one-click setup
└── README.md
| Method | Endpoint | Description |
|---|---|---|
| POST | /api/v1/auth/register |
Create account |
| POST | /api/v1/auth/login |
Login, get JWT |
| GET | /api/v1/auth/me |
Get current user |
| GET | /api/v1/projects |
List projects |
| POST | /api/v1/projects |
Create project |
| POST | /api/v1/projects/{id}/datasets |
Upload dataset (parsed + quality-checked in background) |
| GET | /api/v1/datasets/{id}/preview |
Preview rows |
| GET | /api/v1/datasets/{id}/quality |
Full data quality report |
| GET | /api/v1/datasets/{id}/report |
Download a branded PDF analytics report |
| POST | /api/v1/conversations |
Start conversation |
| POST | /api/v1/conversations/{id}/chat |
Send message — streams tokens, chart, and model-download metadata over SSE |
| GET | /api/v1/health/llm |
Groq health check |
Additional ML-specific endpoints live in app/api/routes/ml.py. Full interactive docs at http://localhost:8000/docs.
Since chat is powered by code generation rather than fixed keywords, phrasing is flexible. Examples that map to specific backend capabilities:
| Ask for... | What runs |
|---|---|
| "histogram of age", "distribution of X" | Plotly histogram |
| "correlation heatmap" | Pearson correlation matrix |
| "pie chart of region", "bar chart of X" | Category chart |
| "detect anomalies" | Isolation Forest + PCA visualization |
| "cluster this data" | K-Means with auto-selected k (elbow + silhouette) |
| "predict charges", "predict X" | Regression model comparison (Ridge / Random Forest / XGBoost) — downloadable model |
| "classify X", "predict whether..." | Classification model comparison (Logistic Regression / Random Forest / XGBoost) — downloadable model |
| "forecast next 30 days" | Prophet time-series forecast |
| Open-ended / "what insights..." | Routed to plain LLM analysis instead of code execution |
ingestion.py is built to survive messy real-world files, not just clean CSVs:
- Auto-detects delimiter (comma, semicolon, tab) and encoding
- Recovers comma-decimal numeric columns (e.g. European exports:
"13,6"→13.6) - Strips stray trailing empty columns from artifacts like
a;b;c;; - Flags likely missing-data sentinel codes (e.g.
-200,9999) that would otherwise silently skew statistics — without guessing and auto-replacing them, since that's a domain judgment call left to the user - Surfaces every parsing decision as a visible warning rather than failing (or worse, succeeding) silently
A few real, non-obvious bugs found and fixed during development — included here because "it works" and "it's correct" aren't always the same thing.
Problem: the ingestion layer tries multiple parsing strategies (comma/semicolon/tab, various quoting rules) before giving up — but on real-world files, every attempt was quietly producing a single garbage column instead of properly parsed data.
Root cause: each strategy explicitly passed low_memory=False, which pandas silently rejects the moment engine="python" is also set — it raises a ValueError immediately, before any real parsing happens. A blanket except: continue was catching that error and hiding it, letting a much worse fallback strategy "succeed" instead.
Fix: only pass low_memory when using the C engine, and surface the real per-strategy exception instead of swallowing it — turning an invisible failure into a debuggable one.
Problem: the automated PDF report generator produced reports with every chart slot empty — no error, just missing images — even though the exact same chart-generation code worked perfectly when run as a standalone script.
Root cause: Kaleido's headless Chromium export calls signal.signal() internally, which Python only permits from the main thread. FastAPI/Starlette runs synchronous route handlers in a background thread pool, not the main thread — so every chart export was failing there with a permissions error that the report generator's broad exception handling was quietly absorbing.
Fix: converted the report-generation route to async def, which runs directly on the event loop (main thread) instead of the thread pool — resolving the conflict without touching the chart code itself.
Problem: the "download trained model" feature produced a working .joblib file — but loading it and running it on new raw data either crashed or silently mispredicted.
Root cause: categorical columns were label-encoded with a fresh LabelEncoder per column at training time, and only the raw fitted estimator was serialized afterward. The encoders themselves — and the exact mapping they'd learned — were discarded, making the saved model impossible to use correctly on new data without reverse-engineering the encoding by hand.
Fix: rebuilt training as a single sklearn.Pipeline (ColumnTransformer + estimator), so preprocessing and the model are serialized together as one self-contained artifact. Verified with adversarial tests — an unseen category, a missing value — to confirm the pipeline degrades gracefully instead of failing.
Problem: some real-world datasets encode missing readings as sentinel numbers (e.g. -200) instead of true nulls — which silently distorts every downstream statistic (means, correlations) without pandas ever flagging the column as having missing data.
Approach: built a generic detector that flags a column when one value is both suspiciously overrepresented (≥3% of rows) and a statistical outlier relative to the rest of that column's distribution (beyond 3×IQR) — a strong signature of a placeholder code. No specific sentinel value is hardcoded, so it generalizes across datasets instead of special-casing one file.
Edit backend/.env (see .env.example):
SECRET_KEY=your-secret-key-change-this
DATABASE_URL=sqlite:///./cortexiq.db
GROQ_API_KEY=your-groq-api-key
GROQ_MODEL=llama-3.1-70b-versatile| Layer | Technology |
|---|---|
| Backend | FastAPI, Python, SQLAlchemy, Alembic |
| Frontend | Next.js, React, Tailwind CSS |
| Database | SQLite (dev) → PostgreSQL (production) |
| LLM | Groq API (Llama 3) |
| Analytics | Pandas, NumPy, SciPy |
| Machine Learning | scikit-learn, XGBoost, Prophet |
| Charts | Plotly |
| Reports | ReportLab + Kaleido (PDF generation) |
| Auth | JWT (python-jose) |
This project is licensed under the MIT License — see LICENSE for details.
Built by Muhammad Ali Ejaz — BS Data Science, IUB Bahawalpur 2023–2027