arxiv-dl

Ruby gem to download papers and metadata from arxiv.org for offline archives.

For each paper it produces a self-contained per-paper folder with:

• The PDF
• The single-page abstract HTML
• The full HTML version (with images and a deduplicated, cross-paper shared assets cache)
• The TeX/LaTeX source tarball, extracted
• Four sidecar metadata files: metadata.md, metadata.yaml, metadata.json, metadata.bib

Installation

gem install arxiv-dl

Or in a Gemfile:

gem 'arxiv-dl'

CLI usage

arxiv-dl <ARXIV_ID_OR_URL> [<ARXIV_ID_OR_URL>...]

Accepted input forms:

Form	Example
Bare ID	2508.16190
Prefixed ID	arXiv:2508.16190
Legacy ID	cs/0002001, alg-geom/9708001
Versioned ID	2508.16190v1
Abstract URL	https://arxiv.org/abs/2508.16190
PDF URL	https://arxiv.org/pdf/2508.16190.pdf
HTML URL	https://arxiv.org/html/2508.16190

Flags

Flag	Description
-i FILE, --input FILE	Read IDs/URLs from FILE, one per line (- for stdin. blanks and # skipped)
-p PATH, --path PATH	Root download directory
--rate-limit SECONDS	Seconds between HTTP requests. 0 disables throttling
-v, --verbose	Print step lines and per-request URL/byte logs to stdout
-q, --quiet	Print nothing to stdout. errors still go to stderr
--version	Print the gem version and exit
-h, --help	Print help and exit

-v and -q are mutually exclusive.

Environment variables

Variable	Effect
ARXIV_DOWNLOAD_PATH	Root download directory (default: $HOME/Downloads/ArXiv_Papers)
ARXIV_RATE_LIMIT	Seconds between HTTP requests (default: 3, per arxiv etiquette. 0 disables)

Precedence: CLI flag > ENV var > default.

Errors and exit status

A target that fails (unrecognized ID, no such paper, HTTP error, network failure) is reported on stderr as <target>: <message>, and the remaining targets still download. Exit status is 0 when every target succeeds and 1 when any fails.

Examples

Download a single paper to the default location:

arxiv-dl 2508.16190

Download to a custom directory:

arxiv-dl --path ~/Archives/arxiv 2508.16190

Download multiple papers, verbose:

arxiv-dl -v 2508.16190 1207.7214 cs/0002001

Download every paper listed in a file, or piped in:

arxiv-dl --input reading-list.txt
cat reading-list.txt | arxiv-dl --input -

Disable rate limiting (when running against a local mirror, etc):

arxiv-dl --rate-limit 0 2508.16190

Output layout

$ARXIV_DOWNLOAD_PATH/                   # default: $HOME/Downloads/ArXiv_Papers
  _shared/                              # cross-paper deduplicated static assets
    arxiv.org/static/...
    cdn.jsdelivr.net/...
  YYYY/MM/DD/<primary_category>/<arxiv-id>-<slug>/
    <arxiv-id>v<N>.pdf
    <arxiv-id>v<N>-abstract.html
    metadata.md                         # YAML frontmatter + Markdown body
    metadata.yaml
    metadata.json
    metadata.bib                        # upstream BibTeX, falls back to synthesized
    html/                               # absent when the paper has no HTML version
      <arxiv-id>v<N>.html               # path-rewritten to local assets
      x1.png, x2.png, ...               # paper-specific images
    src/                                # absent for PDF-only submissions
      *.tex, *.bbl, ...                 # extracted from /src/<id>v<N>

An unversioned ID (2508.16190) archives the latest version. A versioned ID (2508.16190v1) archives that version.

A paper with a single archived version v1 is kept flat, as above. When a paper has more than one version, each version gets its own v<N>/ folder with the same contents:

  YYYY/MM/DD/<primary_category>/<arxiv-id>-<slug>/
    v1/
      <arxiv-id>v1.pdf
      ...
    v2/
      <arxiv-id>v2.pdf
      ...

Archiving a second version of a flat paper first moves the existing files into v<N>/ (rewriting html/ links into _shared/ for the new depth). A paper whose latest version is v2 or later starts out in v<N>/ folders.

Each version downloads into a sibling .partial folder and is renamed into place only when every file succeeded. Re-running skips versions already archived and retries interrupted ones from scratch.

YYYY/MM/DD is the original submission date. <primary_category> is from the paper's metadata (cs.CL, math.NT, etc). <slug> is derived from the paper title (Unicode → ASCII, hyphenated, truncated to 80 chars at a word boundary).

For legacy IDs containing / (e.g. cs/0002001), the slash is replaced with - in the directory name and file names (cs-0002001-.../cs-0002001v1.pdf).

Library usage

require 'arxiv/downloader'

identifier = Arxiv::Downloader::Identifier.new '2508.16190'
client     = Arxiv::Downloader::Client.new                # 3-second rate limit by default
path       = Arxiv::Downloader::Archive.new(identifier, root: '/tmp/papers', client: client).run
# => "/tmp/papers/2025/08/22/cs.CL/2508.16190-comicscene154-a-scene-dataset-for-comic-analysis"

Development

script/setup    # install dependencies
script/test     # run specs and rubocop
script/console  # interactive prompt

Specs run offline against recorded fixtures in spec/fixtures/http/. To check those fixtures against the live arxiv API, run:

ARXIV_LIVE=1 script/test

License

MIT — see LICENSE.md.

Code of Conduct

This project follows the Contributor Covenant 3.0 — see CODE_OF_CONDUCT.md.