Repositories list

• form2request

  Public
  Python library to build HTTP requests out of HTML forms

  Python

  •

  BSD 3-Clause "New" or "Revised" License

  •44 forks•55 stars•22 issues•22 pull requests•Updated Jun 13, 2026Jun 13, 2026

• scrapy

  Public
  Scrapy, a fast high-level web crawling & scraping framework for Python.

  pythoncrawlerframework+ 5

  pythoncrawlerframeworkscrapingcrawlingweb-scrapinghacktoberfestweb-scraping-python

  Python

  •

  BSD 3-Clause "New" or "Revised" License

  •12k12k forks•62k62k stars•430430 issues•189189 pull requests•Updated Jun 12, 2026Jun 12, 2026

• protego

  Public
  A pure-Python robots.txt parser with support for modern conventions.

  pythonrobots-txthacktoberfest+ 1

  pythonrobots-txthacktoberfestrobots-parser

  DIGITAL Command Language

  •

  BSD 3-Clause "New" or "Revised" License

  •3030 forks•8888 stars•77 issues•00 pull requests•Updated Jun 11, 2026Jun 11, 2026

• sphinx-scrapy

  Public
  Sphinx extension for documentation in the Scrapy ecosystem

  Python

  •

  BSD 3-Clause "New" or "Revised" License

  •11 fork•11 star•00 issues•00 pull requests•Updated Jun 11, 2026Jun 11, 2026

• w3lib

  Public
  Python library of web-related functions

  pythonhacktoberfest

  pythonhacktoberfest

  Python

  •

  BSD 3-Clause "New" or "Revised" License

  •113113 forks•419419 stars•1111 issues•77 pull requests•Updated Jun 10, 2026Jun 10, 2026

• scrapyd

  Public
  A service daemon to run Scrapy spiders

  Python

  •

  BSD 3-Clause "New" or "Revised" License

  •578578 forks•3.1k3.1k stars•66 issues•00 pull requests•Updated Jun 8, 2026Jun 8, 2026

• itemadapter

  Public
  Common interface for data container classes

  pythonmetadatapython3+ 4

  pythonmetadatapython3scrapyhacktoberfestpython-dataclassespython-attrs

  Python

  •

  BSD 3-Clause "New" or "Revised" License

  •1313 forks•7070 stars•99 issues•22 pull requests•Updated May 6, 2026May 6, 2026

• scurl

  Public
  Performance-focused replacement for Python urllib

  pythoncythonchromium+ 2

  pythoncythonchromiumgurlurlparse

  Python

  •

  Apache License 2.0

  •66 forks•2121 stars•1010 issues•11 pull request•Updated Apr 13, 2026Apr 13, 2026

• scrapyd-client

  Public
  Command line client for Scrapyd server

  Python

  •

  BSD 3-Clause "New" or "Revised" License

  •145145 forks•773773 stars•55 issues•00 pull requests•Updated Feb 27, 2026Feb 27, 2026

• scrapy-lint

  Public
  A linter for Scrapy projects.

  linterlinter-pluginscrapy+ 2

  linterlinter-pluginscrapyflake8flake8-plugin

  Python

  •

  MIT License

  •44 forks•2222 stars•4242 issues•00 pull requests•Updated Feb 25, 2026Feb 25, 2026

• itemloaders

  Public
  Library to populate items using XPath and CSS with a convenient API

  hacktoberfest

  hacktoberfest

  Python

  •

  BSD 3-Clause "New" or "Revised" License

  •1616 forks•4949 stars•1818 issues•44 pull requests•Updated Jan 29, 2026Jan 29, 2026

• queuelib

  Public
  Collection of persistent (disk-based) and non-persistent (memory-based) queues for Python

  pythonpython3persistent+ 3

  pythonpython3persistenthacktoberfestqueuesnon-persistent

  Python

  •

  BSD 3-Clause "New" or "Revised" License

  •5555 forks•299299 stars•44 issues•22 pull requests•Updated Jan 29, 2026Jan 29, 2026

• parsel

  Public
  Parsel lets you extract data from XML/HTML documents using XPath or CSS selectors

  csspythonxml+ 5

  csspythonxmlscrapingselectorsxpathlxmlhacktoberfest

  Python

  •

  BSD 3-Clause "New" or "Revised" License

  •161161 forks•1.3k1.3k stars•3131 issues•1212 pull requests•Updated Jan 29, 2026Jan 29, 2026

• cssselect

  Public
  CSS Selectors for Python

  csspythonselectors+ 1

  csspythonselectorshacktoberfest

  Python

  •

  Other

  •6464 forks•309309 stars•1818 issues•55 pull requests•Updated Jan 29, 2026Jan 29, 2026

• scrapy-bench

  Public
  A CLI for benchmarking Scrapy.

  pythonweb-crawlerscrapy+ 3

  pythonweb-crawlerscrapycommand-line-toolbenchmark-suitescrapy-bench

  Python

  •

  MIT License

  •1515 forks•3232 stars•66 issues•11 pull request•Updated Jun 28, 2025Jun 28, 2025

• scrapy.org

  Public archive
  The scrapy.org website (old code)

  htmlhacktoberfest

  htmlhacktoberfest

  HTML

  •143143 forks•6666 stars•11 issue•11 pull request•Updated May 8, 2025May 8, 2025

• loginform

  Public
  Fill HTML login forms automatically

  Python

  •8484 forks•279279 stars•1111 issues•33 pull requests•Updated Apr 24, 2024Apr 24, 2024

• xtractmime

  Public
  https://mimesniff.spec.whatwg.org/ implementation for Python

  Python

  •

  BSD 3-Clause "New" or "Revised" License

  •22 forks•1313 stars•00 issues•00 pull requests•Updated Jan 16, 2024Jan 16, 2024

• quotesbot

  Public
  This is a sample Scrapy project for educational purposes

  Python

  •

  MIT License

  •782782 forks•1.4k1.4k stars•22 issues•77 pull requests•Updated Nov 29, 2023Nov 29, 2023

• booksbot

  Public
  A crawler for http://books.toscrape.com

  Python

  •998998 forks•4242 stars•00 issues•44 pull requests•Updated Aug 8, 2023Aug 8, 2023

• scrapely

  Public
  A pure-python HTML screen-scraping library

  HTML

  •273273 forks•1.9k1.9k stars•2626 issues•66 pull requests•Updated Apr 4, 2022Apr 4, 2022

• scrapy-itemloader

  Public archive
  [Archived] Library to populate Scrapy items using XPath and CSS with a convenient API

  Python

  •

  BSD 3-Clause "New" or "Revised" License

  •77 forks•77 stars•22 issues•00 pull requests•Updated May 5, 2020May 5, 2020

• url-chromium

  Public
  url component from Chromium source code, forked from https://chromium.googlesource.com/chromium/src/url

  chromiumgurl

  chromiumgurl

  C++

  •

  BSD 3-Clause "New" or "Revised" License

  •22 forks•44 stars•00 issues•00 pull requests•Updated Aug 7, 2018Aug 7, 2018

• base-chromium

  Public
  base component forked from Chromium source https://chromium.googlesource.com/chromium/src/base/

  C++

  •

  BSD 3-Clause "New" or "Revised" License

  •33 forks•88 stars•00 issues•00 pull requests•Updated Jul 31, 2018Jul 31, 2018

• dirbot

  Public
  Scrapy project to scrape public web directories (educational) [DEPRECATED]

  Python

  •1k1k forks•1.6k1.6k stars•00 issues•00 pull requests•Updated Oct 27, 2017Oct 27, 2017

• scrapy-bench-speedcenter

  Public
  Codespeed for scrapy-bench

  Python

  •22 forks•22 stars•00 issues•00 pull requests•Updated Aug 28, 2017Aug 28, 2017

• pypydispatcher

  Public
  A fork of http://pydispatcher.sourceforge.net/ with PyPy support

  Python

  •

  Other

  •44 forks•1616 stars•11 issue•00 pull requests•Updated Jul 3, 2017Jul 3, 2017

• slybot

  Public
  5555 forks•224224 stars•55 issues•00 pull requests•Updated Apr 27, 2015Apr 27, 2015

• gsoc2014-integration-tests

  Public
  GSoC2014 - Scrapy Integration tests project

  Shell

  •33 forks•33 stars•00 issues•00 pull requests•Updated Mar 18, 2014Mar 18, 2014