---
id: scrapy-playwright
version: "0.0.48"
license: BSD-3-Clause
license_treatment: permissive
maintenance: active
---

# scrapy-playwright — Playwright integration for Scrapy

License: permissive · Maintenance: active · Downloads: 1.4M/mo

## What it is and what it does

scrapy-playwright is a Scrapy download handler that replaces the default HTTP/HTTPS handler to route requests through Playwright, a headless browser automation library. It lets you scrape JavaScript-rendered pages while keeping Scrapy's request scheduling, middleware, and item pipeline intact—requests marked with `meta={"playwright": True}` run through the browser, others use the standard handler.

The package requires Scrapy ≥2.7 (for asyncio support) and Python ≥3.10. After pip install, you must run `playwright install` to download browser binaries. Configuration happens through Scrapy settings: you register the handler in `DOWNLOAD_HANDLERS`, set the Twisted asyncio reactor, and optionally tune browser launch options, browser type (chromium/firefox/webkit), or connect to a remote browser via Chrome DevTools Protocol or WebSocket.

Use it for:

- Scrape single-page applications (SPAs) and client-rendered content that requires JavaScript execution to populate the DOM.
- Handle pages with dynamic loading, infinite scroll, or AJAX-driven content that plain HTTP requests cannot capture.
- Test or monitor web applications that depend on browser-specific behavior, user-agent matching, or JavaScript-based authentication.
- Integrate browser automation into large-scale scraping pipelines without abandoning Scrapy's architecture and concurrency model.
- Connect to remote browser instances (via CDP or WebSocket) for distributed or containerized scraping workflows.

## Worth the install?

AI-flagged interpretation of the facts above — verify before relying.

Integrates Playwright browser automation into Scrapy's download pipeline, enabling JavaScript-heavy pages to be scraped while maintaining Scrapy's standard request scheduling and processing workflow.

Yes, if you need to scrape JavaScript-heavy sites within a Scrapy project. The integration is clean, maintenance is active, and the license is permissive. Install friction is low (pure Python, two deps), but you must meet the version floor (Python ≥3.10, Scrapy ≥2.7) and run `playwright install` for browser binaries. No known vulnerabilities. Not worth it if your target sites are server-rendered or if you prefer a standalone browser automation tool over Scrapy integration.

## Install

pip install scrapy-playwright
uv add scrapy-playwright
poetry add scrapy-playwright

## Installing scrapy-playwright

Before you install: Low friction: pure Python wheel with only two runtime dependencies (scrapy and playwright). Actively maintained with a recent release 35 days ago. Requires Python ≥3.10 and Scrapy ≥2.7, plus manual browser installation via `playwright install` after pip install.

License in practice: BSD-3-Clause permissive license allows commercial and private use with minimal restrictions; attribution required.

Quickstart:

pip install scrapy-playwright
playwright install

# In settings.py:
DOWNLOAD_HANDLERS = {
    "https": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
}
TWISTED_REACTOR = "twisted.internet.asyncioreactor.AsyncioSelectorReactor"

# In spider:
import scrapy
class MySpider(scrapy.Spider):
    async def start(self):
        yield scrapy.Request("https://example.com", meta={"playwright": True})

Requires Python ≥3.10, Scrapy ≥2.7, and manual browser binary installation via `playwright install`. On Windows, requires special asyncio event loop configuration.

Verify before relying:

- Whether the package supports persistent browser contexts or connection pooling for performance at scale.
- Performance overhead compared to regular Scrapy downloads when JavaScript execution is not needed.
- Compatibility with Scrapy middleware that may not expect asyncio-based handlers.

## Package facts

- License: BSD-3-Clause (permissive)
- Python support: supports_current
- Install friction: low
- Maintenance: active
- Downloads: 1.4M/month (top 5,000 on PyPI)
- Known vulnerabilities: none known

## Tags

scrapy javascript rendering, playwright scrapy integration, browser automation web scraping, scrapy dynamic content handler, headless browser scrapy plugin, web-scraping, browser-automation, javascript-rendering

[View on SkillFed](https://skillfed.io/packages/scrapy-playwright) · [View on PyPI](https://pypi.org/project/scrapy-playwright/)