crawlee

What it is and what it does

Crawlee is a Python framework for building web scrapers and crawlers that abstracts away the complexity of HTTP requests, browser automation, parallel execution, and data persistence. It offers two main crawler types: BeautifulSoupCrawler for fast HTML parsing via HTTP, and PlaywrightCrawler for JavaScript-heavy sites that require a headless browser. Both crawlers handle retries, proxy rotation, session management, and request routing through a decorator-based handler system, storing results in a pluggable storage backend.

The library is built on asyncio for efficient concurrent crawling and includes type hints throughout for IDE support and static analysis. It targets developers who want reliable, maintainable scrapers without building infrastructure from scratch—handling bot detection evasion, URL queuing, error recovery, and data export out of the box. Core dependencies are minimal; optional extras (beautifulsoup, playwright) are installed only when needed.

Use it for:

• Extract structured data from static HTML pages using BeautifulSoupCrawler for high-throughput scraping without browser overhead.
• Scrape JavaScript-rendered content by using PlaywrightCrawler to interact with dynamic pages and wait for content generation.
• Build a recursive web crawler that discovers and follows links across a domain with automatic URL queuing and deduplication.
• Rotate proxies and manage sessions automatically to avoid IP blocking and bot detection when scraping protected sites.
• Export crawled data to machine-readable formats (JSON, CSV) with persistent storage for fault-tolerant long-running jobs.

Worth the install?

AI-flagged interpretation of the facts on this page — verify before relying

Crawlee is a web scraping and browser automation library that handles both HTTP-based and headless browser crawling with built-in parallel execution, proxy rotation, request routing, and persistent URL queuing.

Yes. Crawlee is actively maintained, has no known vulnerabilities, low install friction, and a permissive license. It solves a real problem (reliable, parallel web scraping) with a modern async design and type hints. Choose it if you need both HTTP and browser-based crawling in one framework; if you only need simple HTTP requests, a lighter library may suffice.

Install

crawlee on PyPI

pip install crawlee

uv add crawlee

poetry add crawlee

Installing crawlee

Before you install

Low friction installation with a pure-wheel distribution. Actively maintained with a release 8 days old and recent commits. Requires Python 3.10 or later. Optional extras (beautifulsoup, playwright) keep core dependencies lean; full-featured install via crawlee[all] adds browser automation capabilities.

License in practice

Apache-2.0 permissive license allows commercial and private use with minimal restrictions; you must include a copy of the license and state significant changes, but there are no copyleft obligations.

Quickstart

pip install crawlee

import asyncio
from crawlee.crawlers import BeautifulSoupCrawler, BeautifulSoupCrawlingContext

async def main():
    crawler = BeautifulSoupCrawler(max_requests_per_crawl=10)
    @crawler.router.default_handler
    async def handler(context: BeautifulSoupCrawlingContext):
        await context.push_data({'url': context.request.url})
    await crawler.run(['https://example.com'])

asyncio.run(main())

Playwright dependencies must be installed separately (playwright install) for browser-based crawling; BeautifulSoup extra required for HTML parsing crawler.

Verify before relying

• Whether impit (a runtime dependency) is a maintained HTTP client or has known issues.
• Performance characteristics under high concurrency or with large-scale crawls.
• Specific anti-bot evasion techniques and their effectiveness against modern protections.

Package facts

Evidence: crawlee-1.9.1-py3-none-any.whl

Keywords: apify, automation, chrome, crawlee, crawler, headless, scraper, scraping

Tags

More Libraries packages