BibSync

Small CLI that scans a BibTeX file for preprint and working-paper entries and replaces them with their published journal versions when available.

Supported sources

arXiv, NBER, SSRN, CEPR, IZA, bioRxiv, medRxiv, OSF, RePEc, and generic @techreport / "working paper" entries.

Entries that don't match any known source are still picked up if they use @techreport/@unpublished or contain "working paper", "discussion paper", "preprint", or "draft" in the note, howpublished, or type fields. To force detection on any entry, add e.g. note = {working paper}. Detection rules live in src/bibsync/parsers/bibtex_preprints.py.

Install

git clone https://github.com/user/BibSync.git
cd BibSync
uv sync

Usage

# Update publications.bib in place (creates a timestamped .backup first)
uv run python -m bibsync check-preprints publications.bib

# Preview without writing
uv run python -m bibsync check-preprints --dry-run publications.bib

# JSON output (for scripts)
uv run python -m bibsync check-preprints --output-format json publications.bib

# Verbose debug logging
uv run python -m bibsync check-preprints --verbose publications.bib

Each processed entry ends up with status auto-update, no-match, or error.

Try it out

The repo ships with example.bib — a sample mixing preprints from several sources (arXiv, NBER, SSRN, bioRxiv) that have since been published in journals with ones that are still preprint-only:

uv run python -m bibsync check-preprints --dry-run example.bib   # inspect only
uv run python -m bibsync check-preprints example.bib             # update in place

Expected: six auto-update and one no-match (DoubleMLDeep).

How it works

BibSync uses Semantic Scholar (S2) and CrossRef to find published versions of preprints.

1. Detect preprint entries — each .bib entry is classified by source (arXiv, NBER, SSRN, etc.) using metadata like archivePrefix, institution, url, doi, ENTRYTYPE, and free-text fields.
2. Batch arXiv lookup (S2) — arXiv entries are sent to S2's /paper/batch endpoint in a single request using the ARXIV:<id> external-ID form. When S2 has merged the preprint with its published version, this one call already returns the journal + DOI. (This step only runs for arXiv entries — other sources skip straight to step 3.)
3. S2 title-search fallback — when the batch only returns a preprint record (or for non-arXiv sources), the tool falls back to S2's /paper/search with the entry title and picks the top result that (a) has a real journal/DOI, (b) has ≥ 0.7 Jaccard title similarity, and (c) has ≥ 0.3 author-name overlap with the original entry.
4. CrossRef enrichment — when S2 gives us a usable DOI, it's looked up against CrossRef to fill in the canonical journal, volume, issue, pages, and year that S2 often lacks. If S2's DOI is itself unusable (arXiv or a working-paper DOI), CrossRef is re-queried by title to find the real journal DOI.
5. CrossRef last-resort fallback — if S2 returns nothing usable at all (e.g. rate limited, or the paper isn't indexed), the tool queries CrossRef directly by title + first-author. A match is accepted only if the returned title clears the 0.7 similarity threshold.
6. Update — matched entries are written back to the .bib file. A timestamped .backup is created first.

Semantic Scholar API key

The S2 API is free, but unauthenticated calls share a single global rate-limit pool — expect frequent 429s, especially on the title-search fallback. An API key gives you a dedicated ~1 req/s quota. By default, rate-limited requests are not retried (RETRY_ON_429 = False in src/bibsync/api/semantic_scholar.py); flip it to True to enable retries with backoff.

1. Request a key. Apply at https://www.semanticscholar.org/product/api#api-key-form. Approval can take a few days.

2. Set the S2_API_KEY environment variable. BibSync picks it up automatically and switches to a tighter 1.1s request spacing.

# One-off (macOS/Linux)
export S2_API_KEY="your-key-here"
uv run python -m bibsync check-preprints publications.bib

# Persistent (zsh)
echo 'export S2_API_KEY="your-key-here"' >> ~/.zshrc

# One-off (Windows PowerShell)
$env:S2_API_KEY="your-key-here"
uv run python -m bibsync check-preprints publications.bib

# Persistent (user-level)
[Environment]::SetEnvironmentVariable("S2_API_KEY", "your-key-here", "User")