{"categories":[{"label":"Libraries","url":"https://skillfed.io/packages/category/software-development-libraries/4"}],"enrichment":{"capability":"Crawlee is a web scraping and browser automation library that handles both HTTP-based and headless browser crawling with built-in parallel execution, proxy rotation, request routing, and persistent URL queuing.","skillfed_tags":["web-scraping","async-crawler","browser-automation"],"use_cases":["Extract structured data from static HTML pages using BeautifulSoupCrawler for high-throughput scraping without browser overhead.","Scrape JavaScript-rendered content by using PlaywrightCrawler to interact with dynamic pages and wait for content generation.","Build a recursive web crawler that discovers and follows links across a domain with automatic URL queuing and deduplication.","Rotate proxies and manage sessions automatically to avoid IP blocking and bot detection when scraping protected sites.","Export crawled data to machine-readable formats (JSON, CSV) with persistent storage for fault-tolerant long-running jobs."],"what_it_does":"Crawlee is a Python framework for building web scrapers and crawlers that abstracts away the complexity of HTTP requests, browser automation, parallel execution, and data persistence. It offers two main crawler types: BeautifulSoupCrawler for fast HTML parsing via HTTP, and PlaywrightCrawler for JavaScript-heavy sites that require a headless browser. Both crawlers handle retries, proxy rotation, session management, and request routing through a decorator-based handler system, storing results in a pluggable storage backend.\n\nThe library is built on asyncio for efficient concurrent crawling and includes type hints throughout for IDE support and static analysis. It targets developers who want reliable, maintainable scrapers without building infrastructure from scratch\u2014handling bot detection evasion, URL queuing, error recovery, and data export out of the box. Core dependencies are minimal; optional extras (beautifulsoup, playwright) are installed only when needed.","worth_installing":"Yes. Crawlee is actively maintained, has no known vulnerabilities, low install friction, and a permissive license. It solves a real problem (reliable, parallel web scraping) with a modern async design and type hints. Choose it if you need both HTTP and browser-based crawling in one framework; if you only need simple HTTP requests, a lighter library may suffice."},"id":"crawlee","links":{"html":"https://skillfed.io/packages/crawlee","md":"https://skillfed.io/packages/crawlee.md","pypi":"https://pypi.org/project/crawlee/"},"maintenance":{"status":"active"},"meta":{"latest_release":"2026-08-06","license_spdx":"Apache-2.0","license_treatment":"permissive","name":"crawlee","python_support":"supports_current","summary":"Crawlee for Python"},"popularity":{"monthly_downloads":899049,"position":4780,"tier":"top_5000"},"security":{"n_vulnerabilities":0},"version":"1.9.1"}