scrapy-playwright

What it is and what it does

scrapy-playwright is a Scrapy download handler that replaces the default HTTP/HTTPS handler to route requests through Playwright, a headless browser automation library. It lets you scrape JavaScript-rendered pages while keeping Scrapy's request scheduling, middleware, and item pipeline intact—requests marked with `meta={"playwright": True}` run through the browser, others use the standard handler.

The package requires Scrapy ≥2.7 (for asyncio support) and Python ≥3.10. After pip install, you must run `playwright install` to download browser binaries. Configuration happens through Scrapy settings: you register the handler in `DOWNLOAD_HANDLERS`, set the Twisted asyncio reactor, and optionally tune browser launch options, browser type (chromium/firefox/webkit), or connect to a remote browser via Chrome DevTools Protocol or WebSocket.

Use it for:

• Scrape single-page applications (SPAs) and client-rendered content that requires JavaScript execution to populate the DOM.
• Handle pages with dynamic loading, infinite scroll, or AJAX-driven content that plain HTTP requests cannot capture.
• Test or monitor web applications that depend on browser-specific behavior, user-agent matching, or JavaScript-based authentication.
• Integrate browser automation into large-scale scraping pipelines without abandoning Scrapy's architecture and concurrency model.
• Connect to remote browser instances (via CDP or WebSocket) for distributed or containerized scraping workflows.

Worth the install?

AI-flagged interpretation of the facts on this page — verify before relying

Integrates Playwright browser automation into Scrapy's download pipeline, enabling JavaScript-heavy pages to be scraped while maintaining Scrapy's standard request scheduling and processing workflow.

Yes, if you need to scrape JavaScript-heavy sites within a Scrapy project. The integration is clean, maintenance is active, and the license is permissive. Install friction is low (pure Python, two deps), but you must meet the version floor (Python ≥3.10, Scrapy ≥2.7) and run `playwright install` for browser binaries. No known vulnerabilities. Not worth it if your target sites are server-rendered or if you prefer a standalone browser automation tool over Scrapy integration.

Install

scrapy-playwright on PyPI

pip install scrapy-playwright

uv add scrapy-playwright

poetry add scrapy-playwright

Installing scrapy-playwright

Before you install

Low friction: pure Python wheel with only two runtime dependencies (scrapy and playwright). Actively maintained with a recent release 35 days ago. Requires Python ≥3.10 and Scrapy ≥2.7, plus manual browser installation via `playwright install` after pip install.

License in practice

BSD-3-Clause permissive license allows commercial and private use with minimal restrictions; attribution required.

Quickstart

pip install scrapy-playwright
playwright install

# In settings.py:
DOWNLOAD_HANDLERS = {
    "https": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
}
TWISTED_REACTOR = "twisted.internet.asyncioreactor.AsyncioSelectorReactor"

# In spider:
import scrapy
class MySpider(scrapy.Spider):
    async def start(self):
        yield scrapy.Request("https://example.com", meta={"playwright": True})

Requires Python ≥3.10, Scrapy ≥2.7, and manual browser binary installation via `playwright install`. On Windows, requires special asyncio event loop configuration.

Verify before relying

• Whether the package supports persistent browser contexts or connection pooling for performance at scale.
• Performance overhead compared to regular Scrapy downloads when JavaScript execution is not needed.
• Compatibility with Scrapy middleware that may not expect asyncio-based handlers.

Package facts

Evidence: scrapy_playwright-0.0.48-py3-none-any.whl

Tags

More Python Modules packages