Deber Search Engine

• Deber is a search engine written in go and works in its own designated terminal. It is designed to be fast, efficient, and scalable, allowing users to search through large datasets with ease.

The system crawls websites in parallel, parses HTML, runs an NLP pipeline, builds an inverted index in PostgreSQL, generates embeddings via HuggingFace, stores them in Qdrant for semantic search, and exposes a REST API with hybrid ranking.

---

Architecture

---

How to Run

Prerequisites

• HF_TOKEN environment variable set
• PostgreSQL running with database deber created
• Qdrant running on localhost:6334 (Docker)

Commands

# Crawl a website
$env:HF_TOKEN="your_token"; go run . crawl https://go.dev

# Search from CLI
$env:HF_TOKEN="your_token"; go run . search "goroutines concurrency"

# Start HTTP API + scheduler
$env:HF_TOKEN="your_token"; go run . serve

API Endpoints

Method	Endpoint	Description
GET	/api/search?q=...&page=1&limit=10	Search with hybrid ranking
POST	/api/crawl	Trigger crawl {"seed_url": "...", "max_pages": 100}
GET	/api/health	Health check (DB + Qdrant)
GET	/api/stats	Index statistics

Key Environment Variables

Variable	Default	Description
HF_TOKEN	(required)	HuggingFace API token
CRAWL_CROSS_DOMAIN	false	Enable cross-domain crawling
CRAWL_MAX_WORKERS	10	Parallel crawler goroutines
CRAWL_MAX_PAGES	1000	Max pages per crawl job
CRAWL_RATE_PER_SEC	2	Requests/sec per domain
RECRAWL_INTERVAL	@every 24h	Re-crawl schedule (env-configurable)
API_PORT	8080	HTTP server port

---