diff --git a/PROGRESS.md b/PROGRESS.md index fd38a26..b2e3968 100644 --- a/PROGRESS.md +++ b/PROGRESS.md @@ -460,66 +460,78 @@ is closed — see [CONTRIBUTING.md](CONTRIBUTING.md). - **Closes:** #82 - **Status:** In review. -### crawler — persist crawl checkpoints to disk (#95) - -- **What:** Adds `src/crawler/checkpoint.rs` and the new `pub mod crawler` - tree. `CheckpointStore::open(path)` reads (or creates) a - `checkpoints.json` keyed by crawler name. `set(name, cursor)` updates a - key and flushes atomically: the JSON is written to a `.json.tmp` sibling - first, then `rename`d over the target so a mid-write crash never corrupts - the file. `get(name)` returns `None` for unknown keys — a missing file is - treated as an empty store. -- **Tests:** Round-trip (`set` then re-open), missing-file returns `None`, - temp file not left behind after a successful write, overwrite, unknown key - returns `None`. -- **By:** kiro -- **Closes:** #95 -- **Status:** Merged. -### crawler — pipe crawled packages into the typosquat scorer (#96) - -- **What:** Adds `watch_packages`, `PackageSource`, `FakeCrawler`, and - `WATCH_CONFIDENCE_THRESHOLD` to `src/pipeline.rs`. `watch_packages` runs a - loop that calls `source.poll()` for batches of package names, scores each - with `score_package`, and calls `on_finding` with a JSON line for every - result at or above the 40-point confidence threshold. Adds a `watch` - subcommand to `src/bin/depradar.rs` (`depradar watch --registry - npm|pypi|crates --min-confidence N`). -- **Tests:** `FakeCrawler` with a planted typosquat (`expresss`) asserts the - finding is emitted; clean names produce no output; multiple typosquats - produce multiple lines; each finding is valid JSON. -- **By:** kiro -- **Closes:** #96 -- **Status:** Merged. +### crawler — shared rate limiter and polite HTTP client + +- **What:** Adds [`src/net.rs`](src/net.rs) with `PoliteClient`, the one + blocking HTTP client every crawler shares. It wraps `reqwest::blocking` + with a descriptive User-Agent (`DEFAULT_USER_AGENT`, names DepRadar and + links the repo — required by crates.io's crawler policy), a request + timeout (`DEFAULT_TIMEOUT`, 15s), and a per-host token-bucket rate limit + of capacity one (`DEFAULT_PER_HOST_RATE`, 1 req/s per host): one token + refills every `per_host_rate`, so requests to the same host are spaced + evenly. `get`/`get_text`/`get_json` wait for a token before sending. +- **Tests:** Unit tests on the bucket (one immediate request, then delay; + refill after a full period); against mockito mock servers, N requests + take at least `(N-1) × rate`, two different hosts don't share a bucket, + invalid URLs are rejected before any request, and non-2xx statuses are + reported. +- **By:** orsar-rita +- **Closes:** #93 +- **Status:** In review. -### crawler — detect maintainer changes on popular packages (#97) - -- **What:** Adds `src/monitor/maintainers.rs` (new `pub mod monitor` tree). - `MaintainerStore` persists `package → (version → maintainer_set)` as - JSON. `check_maintainers(store, package, version, maintainers)` returns a - `Vec` — one entry per new maintainer that has never - been seen in any previous version of that package. The first observation is - always baseline-only (no finding). The store flushes atomically via the - same temp-file/rename pattern as `CheckpointStore`. -- **Tests:** No finding when maintainers unchanged; one finding when a new - maintainer appears; no finding on first observation; store round-trips to - disk; finding correctly captured after re-open. -- **By:** kiro -- **Closes:** #97 -- **Status:** Merged. +### crawler — PyPI new-package crawler + +- **What:** Adds [`src/crawler/pypi.rs`](src/crawler/pypi.rs). + `PypiCrawler::poll(PypiFeed)` fetches `https://pypi.org/rss/packages.xml` + (new projects) or `/rss/updates.xml` (new releases) through + `PoliteClient`, parses the RSS 2.0 body with a small hand-rolled + ``/``/``/`<link>`/`<pubDate>` extractor (`parse_feed`, + XML entities unescaped), and dedupes against a per-feed `Checkpoint` — + the GUID and publish time of the newest item seen last poll. Items are + reported newest-first; re-polling an unchanged feed yields nothing, and + when the checkpoint GUID ages out of the feed, RFC 822 publish times + (parsed to epoch seconds, no date crate) take over the comparison. +- **Tests:** Fixture RSS bodies cover item order, entity unescaping, + empty feeds, malformed (non-RSS) bodies, epoch conversion of known + dates, checkpoint dedupe (same feed twice, new item on top, GUID aged + out), and two-poll dedupe plus per-feed checkpoints against a mockito + server. +- **By:** orsar-rita +- **Closes:** #91 +- **Status:** In review. -### crawler — watch popular packages for newly added install scripts (#98) - -- **What:** Adds `src/monitor/scripts.rs`. `ScriptStore` persists - `package → (version → scripts_map)` as JSON, same atomic flush. - `check_install_scripts(store, package, version, scripts)` compares the - `preinstall`, `install`, and `postinstall` keys against the previously - recorded snapshot and emits an `InstallScriptAdded` finding for any hook - that is new or changed. The first observation is baseline-only. -- **Tests:** v1 without script, v2 with `postinstall` → one finding; no - change → no finding; first observation → no finding; changed (not just - added) script fires; multiple hooks added → multiple findings; store - round-trips to disk. -- **By:** kiro -- **Closes:** #98 -- **Status:** Merged. +### crawler — crates.io new-crate crawler + +- **What:** Adds [`src/crawler/crates.rs`](src/crawler/crates.rs). + `CratesCrawler::crawl_new` pages + `GET /api/v1/crates?sort=new&per_page=100&page=N` through `PoliteClient` + and stops when it reaches the `Checkpoint` crate (the newest crate seen + last crawl), a short page, or a `max_pages` safety net — so a lost + checkpoint can't walk the whole registry. Policy compliance is documented + in the module docs: the User-Agent and the ≤1 req/s limit come from + `PoliteClient`, and this crawler has no other HTTP path. +- **Tests:** Mocked tests cover the API page shape, paging until the + checkpoint crate on page 2, dedupe across two crawls, the `max_pages` + cap, and stopping on a short page. +- **By:** orsar-rita +- **Closes:** #92 +- **Status:** In review. + +### registry — npm package metadata fetch + +- **What:** Adds [`src/registry/npm.rs`](src/registry/npm.rs). + `fetch_npm_metadata(name)` fetches `https://registry.npmjs.org/<name>` + via `PoliteClient` and reduces the document to + `PackageMeta { name, maintainers, created, modified, repository, + latest, versions_count }`. Scoped names are URL-encoded (`/` → `%2F`), + so `@types/node` is fetched from `/@types%2Fnode`; `repository` accepts + both the bare-string and object form the registry serves. A + `fetch_npm_metadata_with` variant takes a caller-supplied client so a + whole crawl shares one rate-limit budget. +- **Tests:** Fixture documents (lodash, `@types/node`) cover both + repository forms, maintainer usernames, missing optional fields, and a + mockito test asserts the encoded path is requested for a scoped name. +- **By:** orsar-rita +- **Closes:** #94 +- **Status:** In review. diff --git a/src/crawler/crates.rs b/src/crawler/crates.rs new file mode 100644 index 0000000..5638c77 --- /dev/null +++ b/src/crawler/crates.rs @@ -0,0 +1,357 @@ +//! crates.io crawler: newly created crates, from the crates.io API (#92). +//! +//! crates.io has no RSS feed, but `GET /api/v1/crates?sort=new` lists the +//! newest crates first, paged by `page` / `per_page` and stable enough +//! (newest-first) to stop at a checkpoint the same way the PyPI crawler +//! does: once we reach a crate we've already seen, everything after it on +//! later pages is older still. +//! +//! # Crawler policy compliance +//! +//! crates.io asks crawlers to (<https://crates.io/policies>): +//! +//! * send a descriptive User-Agent naming the tool and a contact — the +//! crawler uses [`crate::net::DEFAULT_USER_AGENT`], which identifies +//! DepRadar and links the repository; +//! * make at most one request per second. +//! +//! Both are enforced in [`PoliteClient`], not here: the rate limit is a +//! per-host token bucket, and this crawler never bypasses it (it has no +//! other HTTP path). Keep both in mind if you raise the crawl rate. +//! +//! The API also returns `meta.new_crates`, the total number of crates that +//! exist; a checkpoint can be seeded from it so the very first crawl doesn't +//! walk the whole history. + +use serde::Deserialize; + +use crate::net::{NetError, PoliteClient}; + +/// Base URL of the crates.io API. +pub const CRATES_IO_API_URL: &str = "https://crates.io"; + +/// Crates fetched per API page. crates.io caps `per_page` at 100. +const PER_PAGE: u32 = 100; + +/// Errors from [`CratesCrawler`]. +#[derive(Debug)] +pub enum CratesError { + /// The HTTP request failed (or came back non-2xx). + Net(NetError), + /// The API response wasn't the JSON shape we expect. + Malformed, +} + +impl std::fmt::Display for CratesError { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + match self { + CratesError::Net(err) => write!(f, "crates.io request failed: {err}"), + CratesError::Malformed => write!(f, "crates.io response is not the expected shape"), + } + } +} + +impl std::error::Error for CratesError { + fn source(&self) -> Option<&(dyn std::error::Error + 'static)> { + match self { + CratesError::Net(err) => Some(err), + CratesError::Malformed => None, + } + } +} + +impl From<NetError> for CratesError { + fn from(err: NetError) -> Self { + CratesError::Net(err) + } +} + +/// One crate as listed by `GET /api/v1/crates`. +#[derive(Debug, Clone, PartialEq, Eq, Deserialize)] +pub struct CrateSummary { + /// The crate name, e.g. `serde`. + pub id: String, + /// Most recently published version, e.g. `1.0.210`. + #[serde(default)] + pub newest_version: Option<String>, + /// RFC 3339 creation timestamp, e.g. `2025-10-15T12:00:00.000000Z`. + #[serde(default)] + pub created_at: Option<String>, + #[serde(default)] + pub updated_at: Option<String>, + #[serde(default)] + pub description: Option<String>, + #[serde(default)] + pub downloads: Option<u64>, + #[serde(default)] + pub recent_downloads: Option<u64>, +} + +/// Response body of `GET /api/v1/crates`. +#[derive(Debug, Deserialize)] +pub struct CratesPage { + pub crates: Vec<CrateSummary>, + /// Present on real responses; optional here so minimal fixtures work. + #[serde(default)] + pub meta: Option<serde_json::Value>, +} + +/// Dedupe state for the crates.io feed: the newest crate already seen. +#[derive(Debug, Clone, Default, PartialEq, Eq)] +pub struct Checkpoint { + newest_crate: Option<String>, +} + +impl Checkpoint { + /// A checkpoint that has already seen `newest_crate` (or nothing, when + /// `None`) — typically restored from a previous crawl's state. + pub fn new(newest_crate: Option<&str>) -> Self { + Checkpoint { + newest_crate: newest_crate.map(str::to_string), + } + } + + /// Name of the newest already-seen crate, when any. + pub fn newest_crate(&self) -> Option<&str> { + self.newest_crate.as_deref() + } + + /// True when `name` is the checkpoint crate — the signal to stop paging. + pub fn reached(&self, name: &str) -> bool { + self.newest_crate.as_deref() == Some(name) + } + + /// Moves the checkpoint to the newest of `newly_seen` (crawl order: + /// newest first). + pub fn advance(&mut self, newly_seen: &[CrateSummary]) { + if let Some(newest) = newly_seen.first() { + self.newest_crate = Some(newest.id.clone()); + } + } +} + +/// Polls `GET /api/v1/crates?sort=new`, paging until it reaches the +/// checkpoint crate (or runs out of crates). +/// +/// Requests go through [`PoliteClient`], which enforces crates.io's crawler +/// policy (descriptive User-Agent, ≤1 req/s) — see the module docs. The +/// per-page cost is therefore ~1 second of wall time per 100 crates; a +/// checkpoint from a previous run keeps steady-state crawls to a page or +/// two. Seed the checkpoint with [`Checkpoint::new`] from a saved +/// newest-crate name to skip history entirely on the first crawl. +pub struct CratesCrawler { + client: PoliteClient, + base_url: String, + checkpoint: Checkpoint, + /// Safety net so a lost checkpoint can't walk all ~180k crates. + max_pages: u32, +} + +impl Default for CratesCrawler { + fn default() -> Self { + Self::new(PoliteClient::default()) + } +} + +impl CratesCrawler { + /// A crawler for the live crates.io API. + pub fn new(client: PoliteClient) -> Self { + Self::with_base_url(client, CRATES_IO_API_URL) + } + + /// A crawler against another base URL, e.g. a mock server in tests. + pub fn with_base_url(client: PoliteClient, base_url: &str) -> Self { + CratesCrawler { + client, + base_url: base_url.trim_end_matches('/').to_string(), + checkpoint: Checkpoint::default(), + max_pages: 50, + } + } + + /// Starts from a restored checkpoint, e.g. the newest crate of the + /// previous crawl, so the first crawl after a restart skips history. + pub fn with_checkpoint(mut self, checkpoint: Checkpoint) -> Self { + self.checkpoint = checkpoint; + self + } + + /// Caps how many pages one `crawl_new` call will fetch (each page is + /// `PER_PAGE` crates). Zero disables the limit. + pub fn with_max_pages(mut self, max_pages: u32) -> Self { + self.max_pages = max_pages; + self + } + + /// Names of the crates created since the last crawl, newest first, + /// then advances the checkpoint to the newest of them. + /// + /// An empty result is normal when either nothing new was published or + /// the checkpoint crate is already on page 1. + pub fn crawl_new(&mut self) -> Result<Vec<CrateSummary>, CratesError> { + let mut newly_seen: Vec<CrateSummary> = Vec::new(); + let mut page = 1u32; + loop { + let url = format!( + "{}/api/v1/crates?sort=new&per_page={PER_PAGE}&page={page}", + self.base_url + ); + let body: CratesPage = self.client.get_json(&url)?; + let page_fetched = u32::try_from(body.crates.len()).unwrap_or(u32::MAX); + if page_fetched == 0 { + break; + } + let mut stop = false; + for krate in body.crates { + if self.checkpoint.reached(&krate.id) { + stop = true; + break; + } + newly_seen.push(krate); + } + let last_page = page_fetched < PER_PAGE; + if stop || last_page || (self.max_pages != 0 && page >= self.max_pages) { + break; + } + page += 1; + } + self.checkpoint.advance(&newly_seen); + Ok(newly_seen) + } +} + +#[cfg(test)] +mod tests { + use std::time::Duration; + + use super::*; + + fn fast_client() -> PoliteClient { + PoliteClient::new( + Duration::from_millis(1), + "DepRadar test agent", + Duration::from_secs(5), + ) + .unwrap() + } + + fn page(names: &[&str]) -> String { + let crates: Vec<String> = names + .iter() + .map(|name| { + format!( + r#"{{"id":"{name}","newest_version":"1.0.0","created_at":"2025-10-15T12:00:00.000000Z","downloads":42}}"# + ) + }) + .collect(); + format!( + r#"{{"crates":[{}],"meta":{{"total_crates":3}}}}"#, + crates.join(",") + ) + } + + #[test] + fn page_deserializes_the_api_shape() { + let page: CratesPage = serde_json::from_str(&page(&["alpha", "beta"])).unwrap(); + assert_eq!(page.crates.len(), 2); + assert_eq!(page.crates[0].id, "alpha"); + assert_eq!(page.crates[0].newest_version.as_deref(), Some("1.0.0")); + assert_eq!(page.crates[0].downloads, Some(42)); + assert!(page.meta.is_some()); + } + + /// Two pages; the checkpoint crate is on page 2, so the crawl stops + /// there and never asks for page 3. + #[test] + fn crawl_new_pages_until_the_checkpoint() { + let mut server = mockito::Server::new(); + let page1: Vec<String> = (1..=100).map(|i| format!("c{i:03}")).collect(); + let page1_refs: Vec<&str> = page1.iter().map(String::as_str).collect(); + let m1 = server + .mock("GET", "/api/v1/crates?sort=new&per_page=100&page=1") + .with_body(page(&page1_refs)) + .expect(1) + .create(); + let m2 = server + .mock("GET", "/api/v1/crates?sort=new&per_page=100&page=2") + .with_body(page(&["x-newer", "seen-before", "older"])) + .expect(1) + .create(); + + let mut crawler = CratesCrawler::with_base_url(fast_client(), &server.url()) + .with_max_pages(50) + .with_checkpoint(Checkpoint::new(Some("seen-before"))); + let new = crawler.crawl_new().unwrap(); + assert_eq!(new.len(), 101, "everything up to the checkpoint is new"); + assert_eq!(new[0].id, "c001"); + assert_eq!(new[100].id, "x-newer"); + assert_eq!( + crawler.checkpoint.newest_crate(), + Some("c001"), + "checkpoint moves to the newest crate seen" + ); + m1.assert(); + m2.assert(); + } + + /// A second crawl over the same data reports nothing new. + #[test] + fn crawl_new_dedupes_against_the_checkpoint() { + let mut server = mockito::Server::new(); + let body = page(&["fresh", "fresh-2"]); + let first = server + .mock("GET", "/api/v1/crates?sort=new&per_page=100&page=1") + .with_body(&body) + .expect(1) + .create(); + let second = server + .mock("GET", "/api/v1/crates?sort=new&per_page=100&page=1") + .with_body(&body) + .expect(1) + .create(); + + let mut crawler = CratesCrawler::with_base_url(fast_client(), &server.url()); + let new = crawler.crawl_new().unwrap(); + assert_eq!(new.len(), 2); + assert!(crawler.crawl_new().unwrap().is_empty()); + first.assert(); + second.assert(); + } + + /// The max-pages safety net stops the crawl even without a checkpoint. + #[test] + fn crawl_new_respects_max_pages() { + let mut server = mockito::Server::new(); + let names: Vec<String> = (1..=100).map(|i| format!("c{i:03}")).collect(); + let refs: Vec<&str> = names.iter().map(String::as_str).collect(); + let body = page(&refs); + // Page 2 would also be full, but max_pages = 1 forbids asking. + let m1 = server + .mock("GET", "/api/v1/crates?sort=new&per_page=100&page=1") + .with_body(&body) + .expect(1) + .create(); + + let mut crawler = + CratesCrawler::with_base_url(fast_client(), &server.url()).with_max_pages(1); + let new = crawler.crawl_new().unwrap(); + assert_eq!(new.len(), 100); + m1.assert(); + } + + /// A short page means the end of the listing: no further requests. + #[test] + fn crawl_new_stops_on_a_short_page() { + let mut server = mockito::Server::new(); + let m1 = server + .mock("GET", "/api/v1/crates?sort=new&per_page=100&page=1") + .with_body(page(&["only-one"])) + .expect(1) + .create(); + + let mut crawler = CratesCrawler::with_base_url(fast_client(), &server.url()); + let new = crawler.crawl_new().unwrap(); + assert_eq!(new.len(), 1); + m1.assert(); + } +} diff --git a/src/crawler/mod.rs b/src/crawler/mod.rs index 74795d1..64ddf72 100644 --- a/src/crawler/mod.rs +++ b/src/crawler/mod.rs @@ -1,7 +1,10 @@ -//! Crawler infrastructure — registry crawlers and supporting utilities. +//! Crawlers that watch package registries for newly published packages. //! -//! The crawler side of DepRadar watches package registries for new packages -//! and new versions of existing packages, then hands them off to the scanner -//! pipeline. +//! Everything network-facing goes through [`crate::net::PoliteClient`], so +//! the crawlers themselves only deal with parsing and checkpointing. +//! +//! * [`pypi`] — PyPI's RSS feeds of new projects and updates (#91). +//! * [`crates`] — crates.io's "newest crates" API (#92). -pub mod checkpoint; +pub mod crates; +pub mod pypi; diff --git a/src/crawler/pypi.rs b/src/crawler/pypi.rs new file mode 100644 index 0000000..42804d4 --- /dev/null +++ b/src/crawler/pypi.rs @@ -0,0 +1,519 @@ +//! PyPI crawler: newly published projects and newly uploaded versions, +//! from PyPI's RSS feeds (#91). +//! +//! PyPI publishes two RSS 2.0 feeds: +//! +//! * <https://pypi.org/rss/packages.xml> — new projects, +//! * <https://pypi.org/rss/updates.xml> — new releases of existing projects. +//! +//! Both are newest-first lists of `<item>`s carrying a stable `<guid>` (the +//! project/release URL), a `<title>`, a `<link>` and an RFC 822 +//! `<pubDate>`. The feeds only hold the most recent few hundred entries, so +//! "what have I already seen" is a [`Checkpoint`]: the GUID and publish time +//! of the newest item we processed last poll. [`PypiCrawler::poll`] yields +//! only the items newer than the checkpoint and then advances it, so +//! re-polling an unchanged feed yields nothing. +//! +//! Parsing is a deliberately small hand-rolled extractor rather than an XML +//! library: the feed shape is fixed and tiny, and the crawler already has +//! plenty of dependencies through `reqwest`. + +use crate::net::{NetError, PoliteClient}; + +/// Base URL of the live PyPI site. +pub const PYPI_BASE_URL: &str = "https://pypi.org"; + +/// Errors from [`PypiCrawler`]. +#[derive(Debug)] +pub enum PypiError { + /// The HTTP request failed (or came back non-2xx). + Net(NetError), + /// The response body wasn't an RSS feed we recognise. + MalformedFeed, +} + +impl std::fmt::Display for PypiError { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + match self { + PypiError::Net(err) => write!(f, "PyPI request failed: {err}"), + PypiError::MalformedFeed => write!(f, "PyPI response is not a recognisable RSS feed"), + } + } +} + +impl std::error::Error for PypiError { + fn source(&self) -> Option<&(dyn std::error::Error + 'static)> { + match self { + PypiError::Net(err) => Some(err), + PypiError::MalformedFeed => None, + } + } +} + +impl From<NetError> for PypiError { + fn from(err: NetError) -> Self { + PypiError::Net(err) + } +} + +/// One entry of either PyPI feed. +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct FeedItem { + /// Stable identity of the item, e.g. `https://pypi.org/project/requests/`. + pub guid: String, + /// Human-readable title, e.g. the project name or `name 1.2.3`. + pub title: Option<String>, + /// Canonical project URL. + pub link: Option<String>, + /// Raw RFC 822 `pubDate` as published. + pub pub_date: Option<String>, +} + +/// The publish time of a feed item, as epoch seconds. +/// +/// PyPI emits `Wed, 15 Oct 2025 12:34:56 GMT`-style dates; we parse the +/// subset that format covers (always GMT) and convert to epoch seconds so +/// items can be ordered without a date library. +fn parse_pub_date(rfc822: &str) -> Option<i64> { + let rest = rfc822.split_once(',').map_or(rfc822, |(_, rest)| rest); + let mut words = rest.split_whitespace(); + let day: i64 = words.next()?.parse().ok()?; + let month = month_number(words.next()?)?; + let year: i64 = words.next()?.parse().ok()?; + let mut hms = words.next()?.split(':'); + let hour: i64 = hms.next()?.parse().ok()?; + let minute: i64 = hms.next()?.parse().ok()?; + let second: i64 = hms.next()?.parse().ok()?; + + // Days since 1970-01-01, Howard Hinnant's days-from-civil algorithm: + // shift the year so March starts it, count the era/day-of-era, and + // subtract the days from 0000-03-01 to 1970-01-01 (719468). + let month = month + 1; // back to 1-12, Jan = 1. + let year = if month <= 2 { year - 1 } else { year }; + let era = year.div_euclid(400); + let yoe = year.rem_euclid(400); + let march_month = month + if month > 2 { -3 } else { 9 }; + let doy = (153 * march_month + 2) / 5 + day - 1; + let doe = yoe * 365 + yoe / 4 - yoe / 100 + doy; + let days = era * 146_097 + doe - 719_468; + Some(days * 86_400 + hour * 3_600 + minute * 60 + second) +} + +fn month_number(name: &str) -> Option<i64> { + Some(match name { + "Jan" => 0, + "Feb" => 1, + "Mar" => 2, + "Apr" => 3, + "May" => 4, + "Jun" => 5, + "Jul" => 6, + "Aug" => 7, + "Sep" => 8, + "Oct" => 9, + "Nov" => 10, + "Dec" => 11, + _ => return None, + }) +} + +/// Dedupe state for one feed: the newest item the crawler has already seen. +#[derive(Debug, Clone, Default, PartialEq, Eq)] +pub struct Checkpoint { + guid: Option<String>, + /// Epoch seconds; derived from the checkpoint item's `pubDate`. + timestamp: Option<i64>, +} + +impl Checkpoint { + /// A checkpoint that has already seen the item with this GUID (or + /// nothing, when `None`) — typically restored from a previous run. + pub fn new(guid: Option<&str>, timestamp: Option<i64>) -> Self { + Checkpoint { + guid: guid.map(str::to_string), + timestamp, + } + } + + /// GUID of the newest already-seen item, when any. + pub fn last_guid(&self) -> Option<&str> { + self.guid.as_deref() + } + + /// Keeps only the items newer than this checkpoint. + /// + /// Feeds are newest-first, so this walks until the checkpoint GUID is + /// reached (that item excluded) and returns everything before it. When + /// the checkpoint GUID is missing from the feed (aged out, or a fresh + /// checkpoint), the publish timestamps decide instead; items with no + /// parseable date fall on the safe side and count as new. + pub fn unseen<'a>(&self, items: &'a [FeedItem]) -> Vec<&'a FeedItem> { + if let Some(guid) = &self.guid { + if items.iter().any(|item| &item.guid == guid) { + return items.iter().take_while(|item| &item.guid != guid).collect(); + } + } + match self.timestamp { + None => items.iter().collect(), + Some(seen) => items + .iter() + .filter(|item| { + item.pub_date + .as_deref() + .and_then(parse_pub_date) + .is_none_or(|at| at > seen) + }) + .collect(), + } + } + + /// Moves the checkpoint to the newest of `newly_seen` (feed order: first). + pub fn advance(&mut self, newly_seen: &[&FeedItem]) { + let Some(newest) = newly_seen.first().copied() else { + return; + }; + self.guid = Some(newest.guid.clone()); + self.timestamp = newest.pub_date.as_deref().and_then(parse_pub_date); + } +} + +/// Which of PyPI's two feeds to poll. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum PypiFeed { + /// New projects: <https://pypi.org/rss/packages.xml>. + NewPackages, + /// New releases of existing projects: <https://pypi.org/rss/updates.xml>. + Updates, +} + +impl PypiFeed { + fn path(self) -> &'static str { + match self { + PypiFeed::NewPackages => "/rss/packages.xml", + PypiFeed::Updates => "/rss/updates.xml", + } + } +} + +/// Text content of the first `<tag>` element inside `block`, attribute +/// strings on the start tag included, e.g. `<guid isPermaLink="false">…`. +fn tag_text(block: &str, tag: &str) -> Option<String> { + let open = format!("<{tag}"); + let start = block.find(&open)?; + let after_open = block[start + open.len()..].find('>')? + start + open.len() + 1; + // Skip self-closing tags. + if block.as_bytes().get(after_open.checked_sub(1)?) == Some(&b'/') { + return None; + } + let close = format!("</{tag}>"); + let end = block[after_open..].find(&close)? + after_open; + Some(unescape_xml(&block[after_open..end])) +} + +/// Unescapes the five XML entities that can appear in package titles/URLs. +fn unescape_xml(text: &str) -> String { + text.replace("<", "<") + .replace(">", ">") + .replace(""", "\"") + .replace("'", "'") + .replace("&", "&") +} + +/// Parses one PyPI RSS 2.0 feed body into its items, in document order +/// (newest first). +/// +/// A feed with no `<item>` at all parses to an empty list — PyPI serves an +/// empty feed rather than an error when nothing new arrived. Anything else +/// (an HTML error page, truncated XML) is still best-effort: items missing +/// a `<guid>` are skipped, since the guid is what the checkpoint dedupes on. +pub fn parse_feed(body: &str) -> Result<Vec<FeedItem>, PypiError> { + let mut items = Vec::new(); + let mut rest = body; + while let Some(start) = rest.find("<item>") { + let after = &rest[start + "<item>".len()..]; + let Some(len) = after.find("</item>") else { + return Err(PypiError::MalformedFeed); + }; + let block = &after[..len]; + rest = &after[len + "</item>".len()..]; + let Some(guid) = tag_text(block, "guid") else { + continue; + }; + items.push(FeedItem { + guid, + title: tag_text(block, "title"), + link: tag_text(block, "link"), + pub_date: tag_text(block, "pubDate"), + }); + } + if items.is_empty() && !body.contains("<item") && !body.contains("<?xml") { + return Err(PypiError::MalformedFeed); + } + Ok(items) +} + +/// Polls PyPI's RSS feeds, keeping a per-feed [`Checkpoint`] so that each +/// poll only reports entries the previous one didn't. +/// +/// Requests go through [`PoliteClient`] (rate limit, User-Agent, timeout). +/// The crawler holds one checkpoint per feed, since the two feeds advance +/// independently. +pub struct PypiCrawler { + client: PoliteClient, + base_url: String, + packages: Checkpoint, + updates: Checkpoint, +} + +impl Default for PypiCrawler { + fn default() -> Self { + Self::new(PoliteClient::default()) + } +} + +impl PypiCrawler { + /// A crawler for the live PyPI feeds. + pub fn new(client: PoliteClient) -> Self { + Self::with_base_url(client, PYPI_BASE_URL) + } + + /// A crawler against another base URL, e.g. a mock server in tests. + pub fn with_base_url(client: PoliteClient, base_url: &str) -> Self { + PypiCrawler { + client, + base_url: base_url.trim_end_matches('/').to_string(), + packages: Checkpoint::default(), + updates: Checkpoint::default(), + } + } + + /// Fetches `feed` and returns only the entries not covered by the + /// checkpoint, then advances the checkpoint to the newest of them. + /// + /// An empty result is normal: nothing new since the last poll. + pub fn poll(&mut self, feed: PypiFeed) -> Result<Vec<FeedItem>, PypiError> { + let url = format!("{}{}", self.base_url, feed.path()); + let body = self.client.get_text(&url)?; + let items = parse_feed(&body)?; + + let checkpoint = match feed { + PypiFeed::NewPackages => &mut self.packages, + PypiFeed::Updates => &mut self.updates, + }; + let unseen = checkpoint.unseen(&items); + let newly_seen: Vec<&FeedItem> = unseen.clone(); + checkpoint.advance(&newly_seen); + Ok(newly_seen.into_iter().cloned().collect()) + } +} + +#[cfg(test)] +mod tests { + use std::time::Duration; + + use super::*; + + /// Trimmed copy of the shape PyPI serves at /rss/packages.xml. + const PACKAGES_FEED: &str = r#"<?xml version="1.0" encoding="UTF-8"?> +<rss version="2.0"> + <channel> + <title>PyPI recent updates + + brand-new-lib + https://pypi.org/project/brand-new-lib/ + https://pypi.org/project/brand-new-lib/ + New project 'brand-new-lib' created + Wed, 15 Oct 2025 12:00:00 GMT + + + another-lib + https://pypi.org/project/another-lib/ + https://pypi.org/project/another-lib/ + New project 'another-lib' created + Tue, 14 Oct 2025 09:30:00 GMT + + + old-lib + https://pypi.org/project/old-lib/ + https://pypi.org/project/old-lib/ + New project 'old-lib' created + Mon, 13 Oct 2025 08:00:00 GMT + + +"#; + + fn fast_client() -> PoliteClient { + PoliteClient::new( + Duration::from_millis(1), + "DepRadar test agent", + Duration::from_secs(5), + ) + .unwrap() + } + + #[test] + fn parse_feed_reads_items_in_document_order() { + let items = parse_feed(PACKAGES_FEED).unwrap(); + assert_eq!(items.len(), 3); + assert_eq!(items[0].guid, "https://pypi.org/project/brand-new-lib/"); + assert_eq!(items[0].title.as_deref(), Some("brand-new-lib")); + assert_eq!( + items[0].link.as_deref(), + Some("https://pypi.org/project/brand-new-lib/") + ); + assert_eq!( + items[0].pub_date.as_deref(), + Some("Wed, 15 Oct 2025 12:00:00 GMT") + ); + assert_eq!(items[2].guid, "https://pypi.org/project/old-lib/"); + } + + #[test] + fn parse_feed_unescapes_xml_entities() { + let feed = r#" + a&b<sneaky> "name" + "#; + let items = parse_feed(feed).unwrap(); + assert_eq!(items[0].guid, "a&b"); + assert_eq!(items[0].title.as_deref(), Some(" \"name\"")); + } + + #[test] + fn parse_feed_rejects_non_rss_bodies() { + assert!(matches!( + parse_feed("500 Internal Server Error"), + Err(PypiError::MalformedFeed) + )); + } + + #[test] + fn parse_feed_accepts_an_empty_feed() { + let items = + parse_feed(r#"x"#) + .unwrap(); + assert!(items.is_empty()); + } + + #[test] + fn pub_dates_convert_to_comparable_epoch_seconds() { + // Known value: 2025-10-15T12:00:00Z == 1760529600. + assert_eq!( + parse_pub_date("Wed, 15 Oct 2025 12:00:00 GMT"), + Some(1_760_529_600) + ); + // The Unix epoch itself. + assert_eq!(parse_pub_date("Thu, 01 Jan 1970 00:00:00 GMT"), Some(0)); + assert_eq!(parse_pub_date("not a date"), None); + } + #[test] + fn checkpoint_filters_already_seen_items() { + let items = parse_feed(PACKAGES_FEED).unwrap(); + let mut checkpoint = Checkpoint::default(); + let first_poll = checkpoint.unseen(&items); + assert_eq!(first_poll.len(), 3); + checkpoint.advance(&first_poll); + assert_eq!( + checkpoint.last_guid(), + Some("https://pypi.org/project/brand-new-lib/") + ); + + // Nothing new: the same feed again yields nothing. + assert!(checkpoint.unseen(&items).is_empty()); + + // A genuinely new GUID on top of the feed is reported. + let mut newer = items.clone(); + newer.insert( + 0, + FeedItem { + guid: "https://pypi.org/project/even-newer-lib/".to_string(), + title: Some("even-newer-lib".to_string()), + link: None, + pub_date: Some("Fri, 17 Oct 2025 09:00:00 GMT".to_string()), + }, + ); + let unseen = checkpoint.unseen(&newer); + assert_eq!(unseen.len(), 1); + assert_eq!(unseen[0].guid, "https://pypi.org/project/even-newer-lib/"); + } + + #[test] + fn checkpoint_falls_back_to_timestamps_when_guid_aged_out() { + let items = parse_feed(PACKAGES_FEED).unwrap(); + let mut checkpoint = Checkpoint::default(); + let first = checkpoint.unseen(&items); + checkpoint.advance(&first); + + // The checkpoint GUID has aged out of the feed; timestamps decide. + let mut aged: Vec = items + .iter() + .filter(|i| i.guid != "https://pypi.org/project/brand-new-lib/") + .cloned() + .collect(); + aged.push(FeedItem { + guid: "https://pypi.org/project/brand-newest/".to_string(), + title: None, + link: None, + pub_date: Some("Thu, 16 Oct 2025 10:00:00 GMT".to_string()), + }); + let unseen = checkpoint.unseen(&aged); + assert_eq!(unseen.len(), 1); + assert_eq!(unseen[0].guid, "https://pypi.org/project/brand-newest/"); + } + + /// The crawler polls a mock server twice; the second poll sees only the + /// item that appeared in between. + #[test] + fn poll_dedupes_against_the_checkpoint() { + let mut server = mockito::Server::new(); + let body_v2 = PACKAGES_FEED + .replace( + "Wed, 15 Oct 2025 12:00:00 GMT", + "Fri, 17 Oct 2025 08:00:00 GMT", + ) + .replace("brand-new-lib", "very-new-lib"); + let first = server + .mock("GET", "/rss/packages.xml") + .with_body(PACKAGES_FEED) + .expect(1) + .create(); + let second = server + .mock("GET", "/rss/packages.xml") + .with_body(body_v2) + .expect(1) + .create(); + + let mut crawler = PypiCrawler::with_base_url(fast_client(), &server.url()); + + let poll = crawler.poll(PypiFeed::NewPackages).unwrap(); + assert_eq!(poll.len(), 3, "first poll reports the whole feed"); + + let poll = crawler.poll(PypiFeed::NewPackages).unwrap(); + assert_eq!(poll.len(), 1, "second poll reports only the new item"); + assert_eq!(poll[0].guid, "https://pypi.org/project/very-new-lib/"); + + first.assert(); + second.assert(); + } + + /// Each feed keeps its own checkpoint. + #[test] + fn packages_and_updates_polls_do_not_share_checkpoints() { + let mut server = mockito::Server::new(); + let packages = server + .mock("GET", "/rss/packages.xml") + .with_body(PACKAGES_FEED) + .expect(1) + .create(); + let updates = server + .mock("GET", "/rss/updates.xml") + .with_body(PACKAGES_FEED) + .expect(1) + .create(); + + let mut crawler = PypiCrawler::with_base_url(fast_client(), &server.url()); + assert_eq!(crawler.poll(PypiFeed::NewPackages).unwrap().len(), 3); + assert_eq!(crawler.poll(PypiFeed::Updates).unwrap().len(), 3); + packages.assert(); + updates.assert(); + } +} diff --git a/src/lib.rs b/src/lib.rs index 56a41b5..beca796 100644 --- a/src/lib.rs +++ b/src/lib.rs @@ -15,8 +15,9 @@ pub mod cve; pub mod ignore; pub mod lockfile; pub mod model; -pub mod monitor; +pub mod net; pub mod pipeline; +pub mod registry; pub mod report; pub mod typosquat; diff --git a/src/net.rs b/src/net.rs new file mode 100644 index 0000000..24c9a41 --- /dev/null +++ b/src/net.rs @@ -0,0 +1,299 @@ +//! Shared polite HTTP client for every crawler. +//! +//! Crawling package registries means talking to servers that ask (or require) +//! callers to be good neighbours, so the rules are implemented once here +//! instead of per crawler: +//! +//! * a descriptive User-Agent that identifies DepRadar and links the project, +//! * a request timeout, so a hung server can't stall a crawl forever, and +//! * a per-host token-bucket rate limit ([`TokenBucket`], capacity one: +//! one token refills every `per_host_rate`, so at most one request per +//! `per_host_rate` per host, keyed by the host of the URL we build). +//! +//! The client is blocking ([`reqwest::blocking`]), matching the rest of the +//! off-chain toolkit; the async OSV helpers behind the `async` feature are +//! the exception, not the rule. + +use std::collections::HashMap; +use std::sync::Mutex; +use std::time::{Duration, Instant}; + +use serde::de::DeserializeOwned; + +/// Default rate limit: one request per second per host. +pub const DEFAULT_PER_HOST_RATE: Duration = Duration::from_secs(1); +/// Default request timeout. +pub const DEFAULT_TIMEOUT: Duration = Duration::from_secs(15); +/// Descriptive User-Agent: what we are, where we come from, how to reach us. +/// +/// crates.io's crawler policy () requires a +/// User-Agent that names the tool and a way to contact its operator; keep +/// this honest if you fork or redeploy the crawler. +pub const DEFAULT_USER_AGENT: &str = concat!( + "DepRadar/", + env!("CARGO_PKG_VERSION"), + " (https://github.com/DepRadarOrg/DepRadar; dependency-security crawler)" +); + +/// Errors from [`PoliteClient`]. +#[derive(Debug)] +pub enum NetError { + /// The URL handed to [`PoliteClient::get`] didn't parse. + InvalidUrl { url: String }, + /// The request itself failed (network error, timeout, TLS, etc). + Request(reqwest::Error), + /// The response body wasn't the JSON shape asked for. + Decode(reqwest::Error), + /// A non-success status code. + Status(reqwest::StatusCode), +} + +impl std::fmt::Display for NetError { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + match self { + NetError::InvalidUrl { url } => write!(f, "invalid URL: {url}"), + NetError::Request(err) => write!(f, "request failed: {err}"), + NetError::Decode(err) => write!(f, "failed to decode response: {err}"), + NetError::Status(status) => write!(f, "unexpected status: {status}"), + } + } +} + +impl std::error::Error for NetError { + fn source(&self) -> Option<&(dyn std::error::Error + 'static)> { + match self { + NetError::Request(err) | NetError::Decode(err) => Some(err), + NetError::InvalidUrl { .. } | NetError::Status(_) => None, + } + } +} + +impl From for NetError { + fn from(err: reqwest::Error) -> Self { + if err.is_decode() { + NetError::Decode(err) + } else { + NetError::Request(err) + } + } +} + +/// A token bucket of capacity one: `tokens` refills at `1 / period` per +/// second, and a request consumes the single token. +/// +/// Capacity one (rather than a burst) is deliberate — registries ask for a +/// steady polite rate, not a burst followed by a stall. +#[derive(Debug)] +struct TokenBucket { + tokens: f64, + last_refill: Instant, +} + +impl TokenBucket { + fn full(now: Instant) -> Self { + TokenBucket { + tokens: 1.0, + last_refill: now, + } + } + + /// Refills for elapsed time and takes a token if one is available, + /// returning how long to sleep until one will be instead. + fn take_or_delay(&mut self, period: Duration, now: Instant) -> Option { + let elapsed = now.saturating_duration_since(self.last_refill); + self.last_refill = now; + self.tokens = (self.tokens + elapsed.as_secs_f64() / period.as_secs_f64()).min(1.0); + if self.tokens >= 1.0 { + self.tokens = 0.0; + None + } else { + Some(Duration::from_secs_f64( + (1.0 - self.tokens) * period.as_secs_f64(), + )) + } + } +} + +/// Blocking HTTP client with a per-host rate limit, a descriptive User-Agent +/// and a request timeout. See the module docs for the politeness rules. +#[derive(Debug)] +pub struct PoliteClient { + http: reqwest::blocking::Client, + /// Minimum spacing between two requests to the same host. + per_host_rate: Duration, + buckets: Mutex>, +} + +impl Default for PoliteClient { + fn default() -> Self { + Self::new(DEFAULT_PER_HOST_RATE, DEFAULT_USER_AGENT, DEFAULT_TIMEOUT) + .expect("static reqwest client config is valid") + } +} + +impl PoliteClient { + /// A client that sends `user_agent`, times out after `timeout`, and never + /// issues more than one request per `per_host_rate` to a given host. + pub fn new( + per_host_rate: Duration, + user_agent: &str, + timeout: Duration, + ) -> Result { + let http = reqwest::blocking::Client::builder() + .user_agent(user_agent) + .timeout(timeout) + .build()?; + Ok(PoliteClient { + http, + per_host_rate, + buckets: Mutex::new(HashMap::new()), + }) + } + + /// GET `url`, waiting for this client's per-host rate limit first. + /// + /// With `require_success`, any non-2xx status becomes + /// [`NetError::Status`]; otherwise the response is returned as-is and + /// the caller inspects the status itself. + pub fn get( + &self, + url: &str, + require_success: bool, + ) -> Result { + self.await_turn(url)?; + let response = self.http.get(url).send().map_err(NetError::Request)?; + let status = response.status(); + if require_success && !status.is_success() { + return Err(NetError::Status(status)); + } + Ok(response) + } + + /// GET `url` and decode the body as `T`, erroring on non-2xx. + pub fn get_json(&self, url: &str) -> Result { + let response = self.get(url, true)?; + response.json().map_err(NetError::Decode) + } + + /// GET `url` and return the body as text, erroring on non-2xx. + pub fn get_text(&self, url: &str) -> Result { + let response = self.get(url, true)?; + response.text().map_err(NetError::Request) + } + + /// Blocks until one request to `url`'s host is allowed, then consumes it. + fn await_turn(&self, url: &str) -> Result<(), NetError> { + let parsed = reqwest::Url::parse(url).map_err(|_| NetError::InvalidUrl { + url: url.to_string(), + })?; + let host = parsed.host_str().unwrap_or("").to_string(); + loop { + let delay = { + let mut buckets = self + .buckets + .lock() + .unwrap_or_else(|poisoned| poisoned.into_inner()); + let now = Instant::now(); + let bucket = buckets + .entry(host.clone()) + .or_insert_with(|| TokenBucket::full(now)); + bucket.take_or_delay(self.per_host_rate, now) + }; + match delay { + None => return Ok(()), + Some(delay) => std::thread::sleep(delay), + } + } + } +} + +#[cfg(test)] +mod tests { + use super::*; + + /// A fast client for mock-server tests: no point waiting a real second. + fn fast_client(rate: Duration) -> PoliteClient { + PoliteClient::new(rate, "DepRadar test agent", Duration::from_secs(5)).unwrap() + } + + #[test] + fn bucket_allows_one_immediate_request_then_delays() { + let start = Instant::now(); + let mut bucket = TokenBucket::full(start); + let period = Duration::from_millis(100); + + assert_eq!(bucket.take_or_delay(period, start), None); + // Immediately after, the bucket is empty: we're told to wait ~period. + let delay = bucket + .take_or_delay(period, start + Duration::from_millis(1)) + .expect("second immediate request must be delayed"); + assert!(delay >= Duration::from_millis(90), "delay was {delay:?}"); + assert!(delay <= period); + } + + #[test] + fn bucket_refills_over_time() { + let start = Instant::now(); + let mut bucket = TokenBucket::full(start); + let period = Duration::from_millis(100); + assert_eq!(bucket.take_or_delay(period, start), None); + // One full period later, the token is back. + assert_eq!( + bucket.take_or_delay(period, start + period), + None, + "a full period of refill must restore the token" + ); + } + + #[test] + fn invalid_url_is_rejected_before_any_request() { + let client = fast_client(Duration::from_millis(1)); + assert!(matches!( + client.get("not a url", true), + Err(NetError::InvalidUrl { .. }) + )); + } + + /// N requests to a mock server take at least `(N - 1) * rate` — the + /// first is immediate (the bucket starts full), every later one waits. + #[test] + fn n_requests_take_at_least_the_expected_minimum_time() { + let mut server = mockito::Server::new(); + let mock = server.mock("GET", "/").expect(4).create(); + + let rate = Duration::from_millis(40); + let client = fast_client(rate); + + let requests = 4; + let started = Instant::now(); + for _ in 0..requests { + let response = client.get(&server.url(), true).unwrap(); + assert_eq!(response.status(), 200); + } + let elapsed = started.elapsed(); + + mock.assert(); + let minimum = rate * u32::try_from(requests - 1).unwrap(); + assert!( + elapsed >= minimum, + "{requests} requests at {rate:?} apart took {elapsed:?}, expected at least {minimum:?}" + ); + } + + /// Different hosts don't share a bucket. + #[test] + fn rate_limit_is_per_host() { + let mut a = mockito::Server::new(); + let mut b = mockito::Server::new(); + let mock_a = a.mock("GET", "/").expect(1).create(); + let mock_b = b.mock("GET", "/").expect(1).create(); + + let client = fast_client(Duration::from_secs(60)); + + // Two different hosts: both immediate, despite the 60s rate. + assert_eq!(client.get(&a.url(), true).unwrap().status(), 200); + assert_eq!(client.get(&b.url(), true).unwrap().status(), 200); + mock_a.assert(); + mock_b.assert(); + } +} diff --git a/src/registry/mod.rs b/src/registry/mod.rs new file mode 100644 index 0000000..32b09df --- /dev/null +++ b/src/registry/mod.rs @@ -0,0 +1,6 @@ +//! Registry metadata lookups for individual packages. +//! +//! * [`npm`] — npm registry metadata: maintainers, publish times, +//! repository, latest version (#94). + +pub mod npm; diff --git a/src/registry/npm.rs b/src/registry/npm.rs new file mode 100644 index 0000000..ae9a7ef --- /dev/null +++ b/src/registry/npm.rs @@ -0,0 +1,296 @@ +//! npm registry metadata: the context a scanner needs to judge a package +//! (#94). +//! +//! A version published an hour ago by a brand-new maintainer with no +//! repository is far more suspicious than one published by a maintainer of +//! a decade-old package. [`fetch_npm_metadata`] pulls exactly the fields +//! that tell that story from `https://registry.npmjs.org/`: +//! maintainers, creation/modification times, repository URL, the latest +//! version and how many versions exist. +//! +//! Scoped names (`@scope/name`) are URL-encoded on the wire — the `/` +//! becomes `%2F`, per the registry's documented URL shape +//! . + +use serde::Deserialize; + +use crate::net::{NetError, PoliteClient}; + +/// Base URL of the public npm registry. +pub const NPM_REGISTRY_URL: &str = "https://registry.npmjs.org"; + +/// Errors from [`fetch_npm_metadata`]. +#[derive(Debug)] +pub enum NpmError { + /// The HTTP request failed (or came back non-2xx). + Net(NetError), + /// The response body wasn't the JSON shape we expect. + Malformed, +} + +impl std::fmt::Display for NpmError { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + match self { + NpmError::Net(err) => write!(f, "npm registry request failed: {err}"), + NpmError::Malformed => write!(f, "npm registry response is not the expected shape"), + } + } +} + +impl std::error::Error for NpmError { + fn source(&self) -> Option<&(dyn std::error::Error + 'static)> { + match self { + NpmError::Net(err) => Some(err), + NpmError::Malformed => None, + } + } +} + +impl From for NpmError { + fn from(err: NetError) -> Self { + NpmError::Net(err) + } +} + +/// One maintainer of a package. +#[derive(Debug, Clone, PartialEq, Eq, Deserialize)] +pub struct Maintainer { + /// npm username, e.g. `reqwest_bot`. + #[serde(default)] + pub username: Option, + #[serde(default)] + pub name: Option, + #[serde(default)] + pub email: Option, +} + +/// Registry metadata for one package, as of the fetch. +#[derive(Debug, Clone, PartialEq, Eq, Deserialize)] +pub struct PackageMeta { + /// Package name exactly as the registry returned it. + pub name: String, + #[serde(default)] + pub maintainers: Vec, + /// RFC 3339 creation time, e.g. `2011-03-14T00:51:29.834Z`. + #[serde(default)] + pub created: Option, + #[serde(default)] + pub modified: Option, + /// Repository URL, when the package declares one. + #[serde(default)] + pub repository: Option, + /// Value of `dist-tags.latest`. + #[serde(default)] + pub latest: Option, + /// Number of published versions. + #[serde(default)] + pub versions_count: usize, +} + +/// The subset of the registry document [`fetch_npm_metadata`] reads. +/// +/// `repository` is loosely typed on purpose: the registry serves either a +/// bare string or an object with a `url` field, depending on the package. +#[derive(Deserialize)] +struct NpmDocument { + name: String, + #[serde(default)] + maintainers: Vec, + #[serde(default)] + time: NpmTimes, + #[serde(default)] + repository: Option, + #[serde(default, rename = "dist-tags")] + dist_tags: DistTags, + #[serde(default)] + versions: serde_json::Map, +} + +#[derive(Default, Deserialize)] +struct NpmTimes { + created: Option, + modified: Option, +} + +#[derive(Default, Deserialize)] +struct DistTags { + #[serde(rename = "latest")] + latest: Option, +} + +/// Percent-encodes a package name for the registry URL path. +/// +/// Scoped names contain a `/`, which must become `%2F`; everything else is +/// left alone — npm names are already URL-safe. +pub fn encode_package_name(name: &str) -> String { + name.replace('/', "%2F") +} + +/// Pulls the registry document for `name` and reduces it to [`PackageMeta`]. +/// +/// Requests go through [`PoliteClient`] (rate limit, User-Agent, timeout), +/// so a loop over many names stays polite. Scoped names are encoded: `@types +/// /node` is fetched from `/@types%2Fnode`. +pub fn fetch_npm_metadata(name: &str) -> Result { + fetch_npm_metadata_at(NPM_REGISTRY_URL, name) +} + +/// Like [`fetch_npm_metadata`], against another registry base URL, e.g. a +/// mock server in tests. +pub fn fetch_npm_metadata_at(base_url: &str, name: &str) -> Result { + let client = PoliteClient::default(); + fetch_npm_metadata_with(&client, base_url, name) +} + +/// Like [`fetch_npm_metadata_at`], with an explicit client — the variant +/// crawlers use, so a whole run shares one rate-limit budget. +pub fn fetch_npm_metadata_with( + client: &PoliteClient, + base_url: &str, + name: &str, +) -> Result { + let url = format!( + "{}/{}", + base_url.trim_end_matches('/'), + encode_package_name(name) + ); + let document: NpmDocument = client.get_json(&url)?; + Ok(PackageMeta::from_document(document)) +} + +impl PackageMeta { + fn from_document(document: NpmDocument) -> Self { + PackageMeta { + name: document.name, + maintainers: document.maintainers, + created: document.time.created, + modified: document.time.modified, + repository: repository_url(document.repository.as_ref()), + latest: document.dist_tags.latest, + versions_count: document.versions.len(), + } + } +} + +/// The registry serves `repository` as either a bare string or an object +/// with a `url` field; accept both. +fn repository_url(repository: Option<&serde_json::Value>) -> Option { + match repository? { + serde_json::Value::String(url) => Some(url.clone()), + serde_json::Value::Object(fields) => fields + .get("url") + .and_then(|url| url.as_str()) + .map(str::to_string), + _ => None, + } +} + +#[cfg(test)] +mod tests { + use super::*; + + /// Trimmed copy of a real registry document (unscoped package). + const LODASH_DOC: &str = r#"{ + "name": "lodash", + "dist-tags": {"latest": "4.17.21"}, + "time": { + "created": "2012-03-20T11:11:54.598Z", + "modified": "2025-09-01T10:00:00.000Z" + }, + "repository": {"type": "git", "url": "git+https://github.com/lodash/lodash.git"}, + "maintainers": [{"name": "jdalton", "email": "john.dalton@example.com"}], + "versions": {"4.17.20": {}, "4.17.21": {}} + }"#; + + /// Trimmed copy of a real scoped-package document; `repository` is the + /// bare-string form and maintainers carry usernames. + const SCOPED_DOC: &str = r#"{ + "name": "@types/node", + "dist-tags": {"latest": "22.7.5"}, + "time": { + "created": "2016-01-08T21:16:30.123Z", + "modified": "2025-10-01T09:30:00.000Z" + }, + "repository": "github:DefinitelyTyped/DefinitelyTyped", + "maintainers": [{"username": "types-bot", "email": "types@example.com"}], + "versions": {"22.7.4": {}, "22.7.5": {}} + }"#; + + #[test] + fn parses_metadata_from_a_registry_document() { + let doc: NpmDocument = serde_json::from_str(LODASH_DOC).unwrap(); + let meta = PackageMeta::from_document(doc); + assert_eq!(meta.name, "lodash"); + assert_eq!(meta.latest.as_deref(), Some("4.17.21")); + assert_eq!(meta.created.as_deref(), Some("2012-03-20T11:11:54.598Z")); + assert_eq!(meta.modified.as_deref(), Some("2025-09-01T10:00:00.000Z")); + assert_eq!( + meta.repository.as_deref(), + Some("git+https://github.com/lodash/lodash.git") + ); + assert_eq!(meta.versions_count, 2); + assert_eq!(meta.maintainers.len(), 1); + assert_eq!(meta.maintainers[0].name.as_deref(), Some("jdalton")); + } + + #[test] + fn parses_a_scoped_package_document() { + let doc: NpmDocument = serde_json::from_str(SCOPED_DOC).unwrap(); + let meta = PackageMeta::from_document(doc); + assert_eq!(meta.name, "@types/node"); + assert_eq!( + meta.repository.as_deref(), + Some("github:DefinitelyTyped/DefinitelyTyped") + ); + assert_eq!(meta.versions_count, 2); + assert_eq!(meta.maintainers[0].username.as_deref(), Some("types-bot")); + } + + #[test] + fn scoped_names_are_url_encoded() { + assert_eq!(encode_package_name("@types/node"), "@types%2Fnode"); + assert_eq!(encode_package_name("lodash"), "lodash"); + } + + /// The fetch hits `/@types%2Fnode` — the `/` in the scoped name must not + /// turn into an extra path segment. + #[test] + fn fetch_uses_the_encoded_path_for_scoped_names() { + let mut server = mockito::Server::new(); + let mock = server + .mock("GET", "/@types%2Fnode") + .with_body(SCOPED_DOC) + .expect(1) + .create(); + + let meta = fetch_npm_metadata_at(&server.url(), "@types/node").unwrap(); + assert_eq!(meta.name, "@types/node"); + mock.assert(); + } + + #[test] + fn fetch_works_for_plain_names() { + let mut server = mockito::Server::new(); + let mock = server + .mock("GET", "/lodash") + .with_body(LODASH_DOC) + .expect(1) + .create(); + + let meta = fetch_npm_metadata_at(&server.url(), "lodash").unwrap(); + assert_eq!(meta.versions_count, 2); + mock.assert(); + } + + #[test] + fn missing_fields_stay_none_rather_than_failing() { + let doc: NpmDocument = serde_json::from_str(r#"{"name": "bare", "versions": {}}"#).unwrap(); + let meta = PackageMeta::from_document(doc); + assert_eq!(meta.name, "bare"); + assert!(meta.maintainers.is_empty()); + assert!(meta.created.is_none()); + assert!(meta.repository.is_none()); + assert!(meta.latest.is_none()); + assert_eq!(meta.versions_count, 0); + } +}