🧭 Paper-Pilot

AI-powered research gap analysis — enter a topic, get a structured report on what the literature covers and what's missing.

Paper-Pilot is a full-stack application that automates the tedious first step of academic research. You give it a topic; a multi-agent AI pipeline scrapes papers from Semantic Scholar, reads them, synthesises findings, and delivers a structured gap-analysis report right in your browser.

---

✨ Features

• 🔍 Automated paper discovery — queries Semantic Scholar for relevant publications
• 📄 Deep reading pipeline — LangGraph agents extract key findings from each paper
• 🧠 Gap analysis — Gemini-powered synthesis identifies what's covered and what's missing
• ⚡ Real-time progress — Server-Sent Events (SSE) stream live stage updates to the UI
• 🗄️ Result caching — repeated queries on the same topic return instantly from Supabase
• 🐳 Docker-first — one command brings up the entire stack
• 🔁 Celery + Redis — optional job queue for scalable, async processing

---

🏗️ Architecture

┌─────────────────────┐        ┌──────────────────────────────────────────┐
│   Next.js Frontend  │  HTTP  │           FastAPI Backend                │
│   (Port 3000)       │◄──────►│           (Port 8000)                    │
│                     │  SSE   │                                          │
│  • Research form    │        │  • POST /api/research   (submit job)     │
│  • Progress stream  │        │  • GET  /api/research/{id} (poll status) │
│  • Gap report view  │        │  • GET  /api/research/{id}/progress(SSE) │
└─────────────────────┘        │  • POST /api/research/{id}/cancel        │
                               └──────────┬───────────────────────────────┘
                                          │
                               ┌──────────▼───────────────────────────────┐
                               │        LangGraph AI Pipeline             │
                               │                                          │
                               │  Scraper → Reader → Analyser → Report   │
                               │  (Google Gemini + Semantic Scholar)      │
                               └──────────┬───────────────────────────────┘
                                          │
                    ┌─────────────────────┼──────────────────────┐
                    │                     │                       │
          ┌─────────▼────────┐  ┌─────────▼────────┐  ┌─────────▼────────┐
          │    Supabase      │  │  Redis (optional) │  │  Celery Worker   │
          │  (job storage &  │  │  (job broker)     │  │  (async jobs)    │
          │   result cache)  │  └──────────────────┘  └──────────────────┘
          └──────────────────┘

---

🛠️ Tech Stack

Layer	Technology
Frontend	Next.js 16, React 19, TypeScript, Tailwind CSS v4
Backend	FastAPI, Python 3.11+, Uvicorn
AI Pipeline	LangGraph, LangChain Core, Google Gemini (google-genai)
Paper Source	Semantic Scholar API
Database	Supabase (PostgreSQL)
Job Queue	Celery + Redis (optional, falls back to BackgroundTasks)
Containerisation	Docker, Docker Compose

---

🚀 Getting Started

Prerequisites

• Docker Desktop (recommended) OR
• Python 3.11+, Node.js 18+, and Redis (for local dev without Docker)

1 — Clone the repository

git clone https://github.com/<your-username>/paper-pilot.git
cd paper-pilot

2 — Set up environment variables

Copy the example files and fill in your keys:

# Backend
cp backend/.env.example backend/.env

# Frontend
cp frontend/.env.local.example frontend/.env.local

backend/.env — required values:

Variable	Description
GEMINI_API_KEY	Google AI Studio API key
SUPABASE_URL	Your Supabase project URL (e.g. https://xxxx.supabase.co)
SUPABASE_KEY	Supabase anon or service_role key
SEMANTIC_SCHOLAR_API_KEY	(Optional) Raises rate limit from 100/month → 100/sec
REDIS_URL	(Optional) e.g. redis://localhost:6379/0 — enables Celery
CACHE_TTL_HOURS	(Optional) Result cache duration, default 24

frontend/.env.local:

NEXT_PUBLIC_API_URL=http://localhost:8000

3 — Set up Supabase

Create a table called research_jobs in your Supabase project with the following SQL:

create table research_jobs (
  id          uuid primary key,
  topic       text not null,
  status      text not null default 'processing',
  papers_analyzed integer,
  result_data jsonb,
  error_message text,
  created_at  timestamptz default now()
);

---

🐳 Running with Docker (Recommended)

# Start all services (Redis + Backend + Celery Worker + Frontend)
docker compose up --build

# Or start only the API + worker (no frontend)
docker compose up --build backend celery-worker

# Scale workers horizontally
docker compose up --scale celery-worker=4

Service	URL
Frontend	http://localhost:3000
Backend API	http://localhost:8000
API Docs (Swagger)	http://localhost:8000/docs

---

💻 Running Locally (Without Docker)

Backend

cd backend

# Create and activate a virtual environment
python -m venv agent_env
agent_env\Scripts\activate        # Windows
# source agent_env/bin/activate   # macOS / Linux

# Install dependencies
pip install -r requirements.txt

# Copy and fill in your .env
cp .env.example .env

# Start the API server
uvicorn app.main:app --reload --port 8000

Optional — start the Celery worker (requires Redis running on port 6379):

celery -A app.worker worker --loglevel=info --concurrency=2

Frontend

cd frontend

npm install
cp .env.local.example .env.local   # set NEXT_PUBLIC_API_URL=http://localhost:8000

npm run dev   # http://localhost:3000

---

📡 API Reference

Method	Endpoint	Description
GET	/	Health check
POST	/api/research	Submit a new research job
GET	/api/research/{job_id}	Poll job status & result
GET	/api/research/{job_id}/progress	SSE stream of pipeline stages
POST	/api/research/{job_id}/cancel	Cancel a running job
GET	/api/metrics/summary	Overall metrics summary
GET	/api/metrics/jobs	Recent job metrics

Full interactive docs: http://localhost:8000/docs

---

🗂️ Project Structure

paper-pilot/
├── backend/
│   ├── app/
│   │   ├── agents/          # LangGraph pipeline nodes
│   │   │   ├── orchestrator.py
│   │   │   ├── scraper.py
│   │   │   └── ...
│   │   ├── models/          # Pydantic schemas
│   │   ├── routes/          # Additional route modules
│   │   ├── services/        # Cache, metrics, etc.
│   │   ├── utils/
│   │   ├── config.py
│   │   ├── main.py          # FastAPI app & endpoints
│   │   └── worker.py        # Celery task definitions
│   ├── tests/
│   ├── Dockerfile
│   ├── requirements.txt
│   └── .env.example
│
├── frontend/
│   ├── src/
│   │   ├── app/             # Next.js App Router pages
│   │   ├── components/      # React components
│   │   ├── hooks/           # Custom React hooks
│   │   └── types/           # TypeScript types
│   ├── Dockerfile
│   ├── package.json
│   └── .env.local.example
│
├── docker-compose.yml
├── .gitignore
└── README.md

---

🧪 Testing

cd backend

# Activate your virtual environment first
pytest

# Run with verbose output
pytest -v

# Run a specific test file
pytest tests/test_pipeline.py

---

🔒 Environment Variable Security

⚠️ Never commit .env or .env.local files.
These files are listed in .gitignore by default.
Always use the .env.example / .env.local.example templates as safe placeholders.

---

🤝 Contributing

1. Fork the repository
2. Create a feature branch: git checkout -b feature/your-feature-name
3. Commit your changes: git commit -m 'feat: add some feature'
4. Push to the branch: git push origin feature/your-feature-name
5. Open a Pull Request

---

📄 License

This project is open source. Feel free to use it as a starting point for your own research tools.