feat-crawlers-polite-client-pypi-crates-npm

[orsar-rita]

Implements the four crawler issues in one branch.

What

• [crawler] Shared rate limiter and polite HTTP client #93 — src/net.rs: shared PoliteClient (blocking reqwest) with a descriptive User-Agent, 15s timeout, and a per-host capacity-one token-bucket rate limit (1 req/s/host). Every crawler routes all HTTP through it.
• [crawler] PyPI crawler for newly published packages #91 — src/crawler/pypi.rs: PypiCrawler polls pypi.org/rss/packages.xml and /rss/updates.xml, parses items with a hand-rolled RSS extractor, and dedupes against a per-feed GUID/timestamp checkpoint (RFC 822 to epoch, no date crate).
• [crawler] crates.io crawler for new crates #92 — src/crawler/crates.rs: CratesCrawler pages GET /api/v1/crates?sort=new until the checkpoint crate, a short page, or a max_pages safety net. Module docs describe crates.io crawler-policy compliance (UA + 1 req/s or less via PoliteClient).
• [crawler] Fetch npm package metadata (maintainers, publish time, repo) #94 — src/registry/npm.rs: fetch_npm_metadata(name) returns PackageMeta { maintainers, created, modified, repository, latest, versions_count }; scoped names URL-encoded (/ becomes %2F); both repository shapes accepted; fetch_npm_metadata_with shares one rate-limit budget across a bulk crawl.

Verification

• cargo fmt --all -- --check: clean
• cargo clippy --workspace --all-targets --locked -- -D warnings: clean
• cargo test --workspace --locked: all green (158 lib tests incl. 24 new, all integration suites pass)
• PROGRESS.md entries added for all four issues.

Closes #91
Closes #92
Closes #93
Closes #94

[drips-wave]

@orsar-rita Great news! 🎉 Based on an automated assessment of this PR, the linked Wave issue(s) no longer count against your application limits.

You can now already apply to more issues while waiting for a review of this PR. Keep up the great work! 🚀

Learn more about application limits