{"categories":[{"label":"Python Modules","url":"https://skillfed.io/packages/category/software-development-libraries-python-modules/5"},{"label":"WWW/HTTP","url":"https://skillfed.io/packages/category/internet-www-http/2"},{"label":"Application Frameworks","url":"https://skillfed.io/packages/category/software-development-libraries-application-frameworks"}],"enrichment":{"capability":"Scrapy is a web scraping framework that extracts structured data from websites using a declarative approach with spiders, pipelines, and middleware.","skillfed_tags":["web-scraping","web-crawling","data-extraction"],"use_cases":["Build a crawler to extract product listings, prices, and reviews from e-commerce sites for price comparison or market analysis.","Scrape news articles, headlines, and metadata from multiple news sources and aggregate them into a database.","Monitor competitor websites for changes in pricing, inventory, or content and trigger alerts or updates.","Collect structured data (job postings, real estate listings, classified ads) from websites that don't offer an API.","Extract links, metadata, and content from a website for SEO analysis or content auditing."],"what_it_does":"Scrapy is a mature, production-grade web scraping framework maintained by Zyte that lets you define spiders to crawl and extract structured data from websites. It handles the machinery of HTTP requests, response parsing, data extraction, and pipeline processing, so you focus on defining what to scrape and how to process it. The framework is built on Twisted for asynchronous networking and includes middleware for handling cookies, retries, redirects, and other HTTP concerns.\n\nYou write spiders as Python classes that define start URLs and parsing logic, then Scrapy manages the crawl queue, concurrency, and output. It's designed for both small one-off scrapes and large-scale production crawlers. The 18 runtime dependencies (including lxml for HTML parsing, cryptography for HTTPS, and w3lib for URL handling) are well-established libraries that handle the heavy lifting of web interaction and data extraction.","worth_installing":"Yes. Scrapy is a stable, actively maintained framework with low install friction, permissive licensing, and a large user base. It is the standard choice for production web scraping in Python. The single known vulnerability (PYSEC-2017-83) warrants verification that it does not affect your use case, but the framework's maturity and ongoing maintenance make it a reliable foundation for scraping projects."},"id":"scrapy","links":{"html":"https://skillfed.io/packages/scrapy","md":"https://skillfed.io/packages/scrapy.md","pypi":"https://pypi.org/project/scrapy/"},"maintenance":{"status":"active"},"meta":{"latest_release":"2026-07-07","license_spdx":"BSD-3-Clause","license_treatment":"permissive","name":"Scrapy","python_support":"supports_current","summary":"A high-level Web Crawling and Web Scraping framework"},"popularity":{"monthly_downloads":3853564,"position":2476,"tier":"top_5000"},"security":{"n_vulnerabilities":1},"version":"2.17.0"}