citation-tracker

A small self-hosted system for tracking a citation count (or any other number) across multiple sources, including sites that actively resist scripted/headless access. Four independent pieces, each usable on its own:

• server/ — a tiny Flask receiver. Accepts POSTed {date, citations, source, status} rows, appends them to a CSV, and serves a comparison dashboard. Everything else in this repo is a client of this.
• scholar/ — a daily cron script that logs a Google Scholar citation count. Public data, no login needed, just a library call.
• neko-scraper/ — the main feature. A self-hosted real browser (m1k1o/neko, Chromium in Docker) that you log into once, by hand, through a web UI — then drive programmatically afterward via the Chrome DevTools Protocol, reusing that exact authenticated session. For sites where a plain scripted request gets blocked or challenge-looped no matter how good the fingerprint is.
• extension/ — a read-only browser extension popup that shows the latest tracked value. Fetches from server/; doesn't scrape anything itself.

Architecture

scholar/ (optional)                    neko-scraper/ (main feature)
--------------------                    -----------------------------
cron runs scholar.py,                   you log in by hand, once,
which writes scholar.csv                 through neko's web UI (WebRTC) —
directly (must run on                    for any site that resists
the same host as server/)                headless/bot traffic
        |                                            |
        |                                cron runs run-scraper.sh, which
        |                                drives that same logged-in
        |                                Chromium session over CDP and
        |                                extracts a value via regex
        |                                            |
        |  POST /log                     POST /log   |
        v                                            v

                 +-------------------------------------------------+
                 |  server/server.py                               |
                 |                                                 |
                 |  POST /log        (append a row to CSV)         |
                 |  GET  /data        (read the CSV as JSON)       |
                 |  GET  /dashboard   (render a comparison chart)  |
                 +-------------------------------------------------+
                                        |
                                        v
                 +----------------------------------------------+
                 |  extension/ (popup, read-only)               |
                 |  or any browser hitting /dashboard directly  |
                 +----------------------------------------------+

Note the one structural difference between the two scrapers: scholar/scholar.py writes straight to the CSV file on disk, so it must run on the same host/filesystem as server/. neko-scraper/ talks over HTTP (POST /log), so it can run anywhere with network access to server/ — including a different machine entirely, which is what its optional proxy-tunnel add-on relies on.

Setup

1. server/ — the receiver

cd server
pip install -r requirements.txt
cp .env.example .env   # set SECRET at minimum
set -a; source .env; set +a
python3 server.py

Run this under whatever process supervisor you like (systemd, etc.) for anything beyond testing. Everything else in this repo needs its URL.

2. scholar/ — Google Scholar (optional)

cd scholar
pip install -r requirements.txt
cp .env.example .env   # set SCHOLAR_ID, and LOG_FILE to match server's SCHOLAR_FILE
set -a; source .env; set +a
python3 scholar.py     # test it once

Then cron it daily. Must run on the same host as server/.

3. neko-scraper/ — the real-browser scraper

Full details, including the non-obvious bugs this works around, are in neko-scraper/README.md. Short version:

cd neko-scraper
cp .env.example .env   # BIND_IP, passwords, TARGET_URL, LOG_ENDPOINT, ...
docker compose up -d
# open http://<BIND_IP>:<WEB_UI_PORT>, log in by hand, once
cd scraper && docker build -t rg-citation-scraper:latest . && cd ..
./run-scraper.sh       # test it once

Then cron run-scraper.sh daily.

4. extension/ — the viewer (optional)

Load extension/ as an unpacked extension (chrome://extensions → Developer mode → Load unpacked). Open the popup, enter your server/ URL and secret, save. It's read-only — no permissions beyond talking to whatever server URL you give it.

Security notes

• server/'s SECRET is checked on /log, /data, and /dashboard (as X-Secret header, or ?token= for the dashboard). Set it.
• neko-scraper's web UI is a live, logged-in browser behind one password — see its README for why it should never be bound to 0.0.0.0 or a public IP.
• Nothing here needs your real name, email, or any account identifier committed anywhere — all of that lives in .env files, which are gitignored.

License

MIT — see LICENSE.