Scrapy

What it is and what it does

Scrapy is a mature, production-grade web scraping framework maintained by Zyte that lets you define spiders to crawl and extract structured data from websites. It handles the machinery of HTTP requests, response parsing, data extraction, and pipeline processing, so you focus on defining what to scrape and how to process it. The framework is built on Twisted for asynchronous networking and includes middleware for handling cookies, retries, redirects, and other HTTP concerns.

You write spiders as Python classes that define start URLs and parsing logic, then Scrapy manages the crawl queue, concurrency, and output. It's designed for both small one-off scrapes and large-scale production crawlers. The 18 runtime dependencies (including lxml for HTML parsing, cryptography for HTTPS, and w3lib for URL handling) are well-established libraries that handle the heavy lifting of web interaction and data extraction.

Use it for:

• Build a crawler to extract product listings, prices, and reviews from e-commerce sites for price comparison or market analysis.
• Scrape news articles, headlines, and metadata from multiple news sources and aggregate them into a database.
• Monitor competitor websites for changes in pricing, inventory, or content and trigger alerts or updates.
• Collect structured data (job postings, real estate listings, classified ads) from websites that don't offer an API.
• Extract links, metadata, and content from a website for SEO analysis or content auditing.

Worth the install?

AI-flagged interpretation of the facts on this page — verify before relying

Scrapy is a web scraping framework that extracts structured data from websites using a declarative approach with spiders, pipelines, and middleware.

Yes. Scrapy is a stable, actively maintained framework with low install friction, permissive licensing, and a large user base. It is the standard choice for production web scraping in Python. The single known vulnerability (PYSEC-2017-83) warrants verification that it does not affect your use case, but the framework's maturity and ongoing maintenance make it a reliable foundation for scraping projects.

Install

scrapy on PyPI

pip install scrapy

uv add scrapy

poetry add scrapy

Installing Scrapy

Before you install

Low install friction with a pure-Python wheel distribution. Actively maintained with a recent release (38 days old) and strong repository signals (63846 stars, last commit 2026-08-14). Supports current Python versions (3.10–3.14) on CPython and PyPy.

License in practice

BSD-3-Clause permissive license allows commercial and private use with minimal restrictions, making it suitable for most production and proprietary projects.

Quickstart

pip install scrapy

import scrapy

class MySpider(scrapy.Spider):
    name = 'myspider'
    start_urls = ['http://example.com']
    
    def parse(self, response):
        yield {'title': response.css('h1::text').get()}

Requires Python 3.10 or later; some runtime dependencies (lxml, cryptography, twisted) may require compilation on systems without pre-built wheels.

Verify before relying

• Whether PYSEC-2017-83 remains exploitable in version 2.17.0 or has been patched.
• Performance characteristics and scalability limits for large-scale scraping operations.
• Memory footprint and resource usage when handling concurrent requests.

Package facts

Evidence: scrapy-2.17.0-py3-none-any.whl

Tags

More Python Modules packages