🌊 DataHarbor — web scrapers that fix themselves

Languages: 🇬🇧 English · 🇷🇺 Русский · 🇺🇦 Українська · 🇪🇸 Español · 🇹🇷 Türkçe

2026-09-21: an external audit of the initial public release found 22 issues plus 4 additional observations, spanning self-healing, the data layer, the CLI/Kubernetes deployment, and security (path traversal, a symlink-following publish step, a backup step that could lie about succeeding). All are fixed and verified — several by live reproduction against a running stack or a throwaway kind cluster, not just code review. Full list with root causes and verification: CHANGELOG.md.

Every scraper eventually breaks: a site redesigns its markup, a selector stops matching, and a pipeline quietly starts returning zero rows. Usually that means a human noticing the gap in a dashboard days later, then hand-editing a parser. DataHarbor's scrapers diagnose their own failures and patch themselves — an AI agent reads the failure signal, proposes a code fix, validates it (AST + a real test run against the target), and applies it. You review the diff like any other commit.

That loop is the Harbor Agent Protocol (HAP), and it's the reason this project exists. Everything else — ClickHouse, Postgres, Qdrant, Dagster, observability — is the batteries-included platform that self-healing needs to actually run in production, not just in a demo.

flowchart LR
    A[Scraper run] -->|403 / 429 / ZERO_ROWS| B[harbor health<br/>detects anomaly]
    B --> C[HAP diagnose<br/>builds failure context]
    C --> D[AI proposes a patch<br/>Claude / GPT / Gemini / DeepSeek / Ollama]
    D --> E[AST validation]
    E -->|invalid| D
    E -->|valid| F[Test run against target]
    F -->|fails| D
    F -->|passes| G[Patch applied<br/>scraper.py updated]
    G --> A

Loading

Why this exists

• Self-healing, not just alerting. harbor health --auto-fix <bundle> closes the loop end to end: detect → diagnose → patch → validate → deploy. Model-agnostic (Claude, GPT, Gemini, DeepSeek, or a local Ollama model) — bring your own key, or none at all.
• Batteries-included data stack. PostgreSQL (pgvector), ClickHouse OLAP, Qdrant vector search, Dagster orchestration, and Grafana observability wired together and running in one harbor up, so a pipeline has somewhere real to land data on day one.
• Built for AI coding agents, not just humans. AGENTS.md is a single agent-agnostic contract (skill routing, architecture, verification rules) that Claude Code, Codex, Gemini CLI, and Antigravity all read the same way — see § AI agents & HAP.

What's open vs. private: the platform (apps/), the CLI, and one end-to-end demo bundle are open-core and live in this repo. Real scraping targets — the actual bundles and site-specific extractors — are private by design (see Architecture below) and installed from your own git repos with harbor bundle install. Cloning this repo gets you a working platform and a working example, not a library of ready-made scrapers.

---

🏛 Architecture: Core / Bundles / Extractors

flowchart TB
    subgraph Core["Core (apps/) — open, in this repo"]
        CLI[harbor CLI]
        Dagster[Dagster orchestration]
        DB[(Postgres · ClickHouse · Qdrant)]
        Obs[Observability + HAP]
    end
    subgraph Bundles["Bundles (bundles/) — one git repo each"]
        B1[fetch → parse → schema → export]
    end
    subgraph Extractors["Extractors (extractors/) — one per source"]
        E1["parse(html) only<br/>no HTTP/proxy"]
    end
    CLI -->|harbor bundle install| Bundles
    Bundles -->|requirements.extractors| Extractors
    Bundles --> DB
    Dagster --> Bundles
    Obs -.->|failure signal| CLI

Loading

• Core (apps/) — open platform runtime: PostgreSQL pgvector, ClickHouse OLAP, Qdrant, SeaweedFS S3, Dagster, Core notifier (Telegram/Slack/webhooks). Optional: uv sync --extra ml, n8n via --with-n8n. Backup CLI stays in Core (harbor backup).
• Bundles (bundles/) — business pipelines (fetch → schema → Dagster → HTML/CSV export). One bundle = one git repository when published.
• Extractors (extractors/) — parse plugins for a single source (parse(html, …) only; no HTTP/proxy). Reused by multiple bundles. Open-core ships the demo_site example; domain extractors are installed from git.

A bundle declares dependencies in manifest.json → requirements.extractors (local id, git URL, or {name, source}). On harbor bundle install, the platform pulls declared extractor sources into extractors/.

---

⚡ Quick Start

1. Install

git clone https://github.com/eugene-panin/DataHarbor.git
cd DataHarbor
./install.sh   # requires uv; installs deps via `uv sync` and links `harbor`

uv sync
uv sync --extra ml   # optional: Whisper / EasyOCR / embeddings / HDBSCAN
uv run harbor --help
uv run harbor doctor   # includes Publish readiness (git identity; gh optional)

2. Start the platform

harbor up

Starts platform services in Kubernetes (Tilt) or Docker Compose (harbor up --compose).

3. Check status

harbor status

4. Try the demo bundle (no API keys, no external network)

Open-core ships a working end-to-end example: a static demo_site fixture (deploy/demo_site/) scraped into PostgreSQL by the demo bundle (bundles/demo/).

harbor bundle run demo    # scrapes the local fixture, writes to PostgreSQL
harbor bundle view demo   # renders an HTML report of the scraped rows

harbor bundle run shells out to dagster asset materialize; you can do the same by hand in the Dagster UI (materialize the demo asset group). Use this bundle's files (fetch.py / scraper.py / db.py / assets.py / exporter.py) as a starting point for your own.

5. See it heal itself

Break the demo on purpose, then let HAP fix it:

# Simulate drift: rename the selector the demo_site extractor looks for
sed -i.bak 's/h1, h2, h3/h1, h3/' extractors/demo_site/extractor.py
harbor bundle run demo                              # now returns 0 rows — a real ZERO_ROWS anomaly
harbor health                                        # flags demo as DEGRADED
harbor health --auto-fix demo --url http://demo_site/   # diagnose, patch, AST-validate, live-verify
harbor bundle run demo                               # rows are back

--url closes the loop for real: after AST validation the patch is applied and live-scraped against that URL. If it still returns zero rows, the failure is fed back to the model for one retry; if that also fails, scraper.py is rolled back to the pre-patch version instead of being left in a broken, "looks patched" state. Without --url the patch only gets an import check — enough to catch a broken reference, not enough to prove the scrape works.

Needs an LLM key in .env — AI_REPAIR_PROVIDER / AI_REPAIR_API_KEY, or AI_REPAIR_PROVIDER=ollama for a local model. No key configured? harbor health --fix demo prints the diagnostic prompt instead so you can see exactly what the agent would work from.

---

📦 Publish: bundles and extractors

Model: 1 plugin = 1 git repo. Staging uses a temp directory (or --workdir); nested .git is never created inside the monorepo.

# New extractor / bundle
harbor extractor new demo_site
harbor bundle new my_leads --extractors demo_site

# Publish a custom extractor first
harbor extractor publish demo_site --dry-run
harbor extractor publish demo_site
# → put in the bundle manifest: git@github.com:<you>/dh-extractor-demo-site.git

# Then publish the bundle
harbor bundle publish my_leads --dry-run
harbor bundle publish my_leads
# or explicitly: --remote git@github.com:ORG/dh-bundle-my_leads.git

Important: harbor bundle publish does not auto-create extractor repos on GitHub. If requirements.extractors still lists local ids/paths (not git URLs), the CLI lists them and suggests harbor extractor publish <id>. A real publish stops; --dry-run only warns. Escape hatch: --allow-local-extractors. The shipped demo_site example is skipped.

Flags: --remote, --workdir, --repo-name, --visibility private|public, --public, --dry-run. Requires git plus user.name / user.email; gh is optional (auto-create a private GitHub repo).

---

🌐 Local web services and ports

Service	Description	Local URL
☸️ Tilt	Local Kubernetes control panel	http://localhost:10350
⚡ Dagster UI	ETL orchestrator and auto-repair dashboard	http://localhost:3000
🚀 ClickHouse	OLAP warehouse for analytics	http://localhost:8123
🧭 Qdrant	Vector search / RAG	http://localhost:6333/dashboard
📊 Grafana	Observability dashboards	http://localhost:3001
🔄 n8n (optional)	No-code automation hub	http://localhost:56780 (harbor up --with-n8n)
🧪 demo_site	Static fixture scraped by the demo bundle	http://localhost:8098

---

🛠 CLI (harbor)

1. Environment diagnostics and control

harbor doctor             # Docker, Tilt, Kind, Python, Git + Publish readiness
harbor health             # Scraper health, zero-row anomalies, SLA
harbor health --auto-fix <bnd> --url <url> # AI remediator: AST-validate + live-verify + rollback on failure
harbor up                 # Kubernetes/Tilt (or --compose)
harbor down
harbor status
harbor uninstall          # Remove with auto-backup to ~/DataHarbor_Backups

2. Master backup and restore

harbor backup create
harbor backup restore <file.tar.gz>
harbor backup list

3. Harbor Agent Protocol (HAP v1.0)

harbor agent-protocol status
harbor agent-protocol diagnose <bnd>
harbor agent-protocol patch <bnd> --code-file <file>
harbor agent-protocol test <bnd>

4. Bundles

harbor bundle list
harbor bundle new <name> [--template default|ml|etl|dagster]
harbor bundle templates
harbor bundle install <src>   # Git / archive / folder; pulls declared extractors
harbor bundle resolve <name>  # Install missing extractors from manifest
harbor bundle pack <name>
harbor bundle publish <name> [--dry-run] [--remote URL] [--allow-local-extractors]
harbor bundle remove <name>
harbor bundle run <name>      # materialize all Dagster assets synchronously (no UI needed)
harbor bundle export / view
harbor bundle validate
harbor bundle doctor [name]   # engines, python deps, extractors, Dagster defs
harbor workspace refresh      # regenerate multi code-location workspace.yaml
harbor workspace show

5. Extractors

harbor extractor list
harbor extractor new <name> [--path DIR]
harbor extractor install <src>
harbor extractor pack <name>
harbor extractor publish <name> [--dry-run] [--remote URL]
harbor extractor resolve <bundle>
harbor extractor validate [name]
harbor extractor remove <name>

---

🤖 AI agents & the Harbor Agent Protocol

AGENTS.md is the single, agent-agnostic contract for this repo — architecture, the bundle/extractor manifest schema, and HAP v1.0's diagnose → patch → validate → test flow. CLAUDE.md and GEMINI.md are thin pointers to it, so Claude Code, Codex, Gemini CLI, and Antigravity all read the same source of truth instead of drifting.

Three scoped skills route work by intent instead of one do-everything prompt:

Skill	Job
dataharbor-bundle-designer	Write or scaffold a new bundle/extractor
dataharbor-bundle-operator	Check health/SLA, run harbor agent-protocol summary
dataharbor-remediator	Diagnose and repair a broken scraper (HAP)

harbor skill install copies them into ~/.claude/skills (and the equivalent path for Codex/Gemini/Antigravity) so any of those tools picks them up automatically.

---

🤝 Contributing

Contributions to the open-core platform are welcome — see CONTRIBUTING.md for dev setup, the test/lint gates CI runs, and PR guidelines. Please also read the Code of Conduct.

Found a security issue? Please report it privately — see SECURITY.md, don't open a public issue.

---

📄 License

MIT License © DataHarbor Core Team