Scrapy
A high-level Web Crawling and Web Scraping framework
Decision gist · record as of 2026-08-14
Yes. Scrapy is a stable, actively maintained framework with low install friction, permissive licensing, and a large user base. It is the standard choice for production web scraping in Python. The single known vulnerability (PYSEC-2017-83) warrants verification that it does not affect your use case, but the framework's maturity and ongoing maintenance make it a reliable foundation for scraping projects.AI-flagged interpretation of the facts on this page — verify before relying
Before you install
- Requires Python 3.10 or later; some runtime dependencies (lxml, cryptography, twisted) may require compilation on systems without pre-built wheels.
- Low install friction with a pure-Python wheel distribution.
- Actively maintained with a recent release (38 days old) and strong repository signals (63846 stars, last commit 2026-08-14).
License · maintenance · safety
BSD-3-Clause (permissive) — BSD-3-Clause permissive license allows commercial and private use with minimal restrictions, making it suitable for most production and proprietary projects.
last release 2026-07-07 (38 days) · last repo commit 2026-08-14 · 63,846 stars
1 known vulnerabilities (OSV.dev, 2026-08-14) · 3,853,564 downloads/mo, #2,476 on PyPI
Alternatives
Verify before relying
pip install scrapy
import scrapy
class MySpider(scrapy.Spider):
name = 'myspider'
start_urls = ['http://example.com']
def parse(self, response):
yield {'title': response.css('h1::text').get()}- Whether PYSEC-2017-83 remains exploitable in version 2.17.0 or has been patched.
- Performance characteristics and scalability limits for large-scale scraping operations.
- Memory footprint and resource usage when handling concurrent requests.
What it is and what it does
Scrapy is a mature, production-grade web scraping framework maintained by Zyte that lets you define spiders to crawl and extract structured data from websites. It handles the machinery of HTTP requests, response parsing, data extraction, and pipeline processing, so you focus on defining what to scrape and how to process it. The framework is built on Twisted for asynchronous networking and includes middleware for handling cookies, retries, redirects, and other HTTP concerns.
You write spiders as Python classes that define start URLs and parsing logic, then Scrapy manages the crawl queue, concurrency, and output. It's designed for both small one-off scrapes and large-scale production crawlers. The 18 runtime dependencies (including lxml for HTML parsing, cryptography for HTTPS, and w3lib for URL handling) are well-established libraries that handle the heavy lifting of web interaction and data extraction.
Use it for
- Build a crawler to extract product listings, prices, and reviews from e-commerce sites for price comparison or market analysis.
- Scrape news articles, headlines, and metadata from multiple news sources and aggregate them into a database.
- Monitor competitor websites for changes in pricing, inventory, or content and trigger alerts or updates.
- Collect structured data (job postings, real estate listings, classified ads) from websites that don't offer an API.
- Extract links, metadata, and content from a website for SEO analysis or content auditing.
Worth the install?
AI-flagged interpretation of the facts on this page. Verify before relying on it.
Yes.
Scrapy is a stable, actively maintained framework with low install friction, permissive licensing, and a large user base. It is the standard choice for production web scraping in Python. The single known vulnerability (PYSEC-2017-83) warrants verification that it does not affect your use case, but the framework's maturity and ongoing maintenance make it a reliable foundation for scraping projects.
Install
scrapy on PyPI
Before you install
Low install friction with a pure-Python wheel distribution. Actively maintained with a recent release (38 days old) and strong repository signals (63846 stars, last commit 2026-08-14). Supports current Python versions (3.10–3.14) on CPython and PyPy.
Requires Python 3.10 or later; some runtime dependencies (lxml, cryptography, twisted) may require compilation on systems without pre-built wheels.
License in practice
BSD-3-Clause permissive license allows commercial and private use with minimal restrictions, making it suitable for most production and proprietary projects.
Quickstart
pip install scrapy
import scrapy
class MySpider(scrapy.Spider):
name = 'myspider'
start_urls = ['http://example.com']
def parse(self, response):
yield {'title': response.css('h1::text').get()}Verify before relying
- Whether PYSEC-2017-83 remains exploitable in version 2.17.0 or has been patched.
- Performance characteristics and scalability limits for large-scale scraping operations.
- Memory footprint and resource usage when handling concurrent requests.
Package facts
| License | BSD-3-Clause permissive |
| Python support | Supports the current Python release >=3.10 |
| Install friction | Low. Pure-Python wheel |
| Runtime dependencies | 18 packagescryptographycssselectdefusedxmlitemadapteritemloaderslxmlpackagingparselprotegopydispatcherpyopensslpypydispatcherqueuelibservice-identitytldextracttwistedw3libzope-interface |
| Maintenance | Actively maintained 38 days since the last release |
| Last repo commit | |
| First released | |
| Downloads | 3,853,564 / month, #2,476 on PyPI 30-day window, as of 2026-08-14 |
| Known vulnerabilities | 1 PYSEC-2017-83 |
| Classifiers | Development Status :: 5 - Production/StableEnvironment :: ConsoleFramework :: ScrapyIntended Audience :: DevelopersOperating System :: OS IndependentProgramming Language :: PythonProgramming Language :: Python :: 3Programming Language :: Python :: 3.10Programming Language :: Python :: 3.11Programming Language :: Python :: 3.12Programming Language :: Python :: 3.13Programming Language :: Python :: 3.14Programming Language :: Python :: Implementation :: CPythonProgramming Language :: Python :: Implementation :: PyPyTopic :: Internet :: WWW/HTTPTopic :: Software Development :: Libraries :: Application FrameworksTopic :: Software Development :: Libraries :: Python Modules |
Evidence: scrapy-2.17.0-py3-none-any.whl
Tags
Let your AI agent find packages like this
Example. Real query, live index.
You found this page by searching. An agent finds it by wishing: SkillFed indexes 14,416 PyPI packages by what they can do, searchable in plain language.
wish › “web scraping framework”
- ScrapyScrapy is a web scraping framework that extracts structured data from…
- scraplingScrapling is a web scraping and crawling framework that handles…
- crawleeCrawlee is a web scraping and browser automation library that handles…
Give your agent the search over MCP, or paste the wish link into any chat.
More Python Modules packages
Converts domain names between Unicode and ASCII-compatible encoding (Punycode) according to IDNA 2008 and Unicode Technical Standard 46, with security validation and broader script coverage than the standard library.
Install it if you work with internationalized domain names, need to validate domains, or use HTTP clients that depend on it transitively.
Setuptools is a Python build backend and package management tool that handles building, distributing, and installing Python packages, including support for C/C++ extension modules.
PyYAML parses and emits YAML 1.1 data format, enabling serialization and deserialization of configuration files and Python objects to and from human-readable YAML text.
Pydantic validates Python data structures against type hints, coercing and checking input at runtime to ensure it matches a declared schema.
Provides reusable metadata objects for use with PEP-593 `typing.Annotated` to express common constraints like bounds, collection sizes, and predicates on types.
Install it if you use or build libraries that need to express type constraints in a standardized, inspectable way—or if you want to annotate your own types with…
Provides runtime tools to inspect and introspect Python type annotations, enabling programmatic examination of type hints at execution time.
See also recipe-scrapers · scrapling · LinkChecker · scrapy-zyte-api · scrapydo · firecrawl · trafilatura · crawlee · scrapinghub · icrawler