scrapy-playwright
Playwright integration for Scrapy
Decision gist · record as of 2026-08-14
Yes, if you need to scrape JavaScript-heavy sites within a Scrapy project. The integration is clean, maintenance is active, and the license is permissive. Install friction is low (pure Python, two deps), but you must meet the version floor (Python ≥3.10, Scrapy ≥2.7) and run `playwright install` for browser binaries. No known vulnerabilities. Not worth it if your target sites are server-rendered or if you prefer a standalone browser automation tool over Scrapy integration.AI-flagged interpretation of the facts on this page — verify before relying
Before you install
- Requires Python ≥3.10, Scrapy ≥2.7, and manual browser binary installation via `playwright install`.
- On Windows, requires special asyncio event loop configuration.
- Low friction: pure Python wheel with only two runtime dependencies (scrapy and playwright).
License · maintenance · safety
BSD-3-Clause (permissive) — BSD-3-Clause permissive license allows commercial and private use with minimal restrictions; attribution required.
last release 2026-07-10 (35 days) · last repo commit 2026-07-23 · 1,440 stars
0 known vulnerabilities (OSV.dev, 2026-08-14) · 1,397,896 downloads/mo, #3,957 on PyPI
Alternatives
Verify before relying
pip install scrapy-playwright
playwright install
# In settings.py:
DOWNLOAD_HANDLERS = {
"https": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
}
TWISTED_REACTOR = "twisted.internet.asyncioreactor.AsyncioSelectorReactor"
# In spider:
import scrapy
class MySpider(scrapy.Spider):
async def start(self):
yield scrapy.Request("https://example.com", meta={"playwright": True})- Whether the package supports persistent browser contexts or connection pooling for performance at scale.
- Performance overhead compared to regular Scrapy downloads when JavaScript execution is not needed.
- Compatibility with Scrapy middleware that may not expect asyncio-based handlers.
What it is and what it does
scrapy-playwright is a Scrapy download handler that replaces the default HTTP/HTTPS handler to route requests through Playwright, a headless browser automation library. It lets you scrape JavaScript-rendered pages while keeping Scrapy's request scheduling, middleware, and item pipeline intact—requests marked with `meta={"playwright": True}` run through the browser, others use the standard handler.
The package requires Scrapy ≥2.7 (for asyncio support) and Python ≥3.10. After pip install, you must run `playwright install` to download browser binaries. Configuration happens through Scrapy settings: you register the handler in `DOWNLOAD_HANDLERS`, set the Twisted asyncio reactor, and optionally tune browser launch options, browser type (chromium/firefox/webkit), or connect to a remote browser via Chrome DevTools Protocol or WebSocket.
Use it for
- Scrape single-page applications (SPAs) and client-rendered content that requires JavaScript execution to populate the DOM.
- Handle pages with dynamic loading, infinite scroll, or AJAX-driven content that plain HTTP requests cannot capture.
- Test or monitor web applications that depend on browser-specific behavior, user-agent matching, or JavaScript-based authentication.
- Integrate browser automation into large-scale scraping pipelines without abandoning Scrapy's architecture and concurrency model.
- Connect to remote browser instances (via CDP or WebSocket) for distributed or containerized scraping workflows.
Worth the install?
AI-flagged interpretation of the facts on this page. Verify before relying on it.
Yes, if you need to scrape JavaScript-heavy sites within a Scrapy project.
The integration is clean, maintenance is active, and the license is permissive. Install friction is low (pure Python, two deps), but you must meet the version floor (Python ≥3.10, Scrapy ≥2.7) and run `playwright install` for browser binaries. No known vulnerabilities. Not worth it if your target sites are server-rendered or if you prefer a standalone browser automation tool over Scrapy integration.
Install
scrapy-playwright on PyPI
Before you install
Low friction: pure Python wheel with only two runtime dependencies (scrapy and playwright). Actively maintained with a recent release 35 days ago. Requires Python ≥3.10 and Scrapy ≥2.7, plus manual browser installation via `playwright install` after pip install.
Requires Python ≥3.10, Scrapy ≥2.7, and manual browser binary installation via `playwright install`. On Windows, requires special asyncio event loop configuration.
License in practice
BSD-3-Clause permissive license allows commercial and private use with minimal restrictions; attribution required.
Quickstart
pip install scrapy-playwright
playwright install
# In settings.py:
DOWNLOAD_HANDLERS = {
"https": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
}
TWISTED_REACTOR = "twisted.internet.asyncioreactor.AsyncioSelectorReactor"
# In spider:
import scrapy
class MySpider(scrapy.Spider):
async def start(self):
yield scrapy.Request("https://example.com", meta={"playwright": True})Verify before relying
- Whether the package supports persistent browser contexts or connection pooling for performance at scale.
- Performance overhead compared to regular Scrapy downloads when JavaScript execution is not needed.
- Compatibility with Scrapy middleware that may not expect asyncio-based handlers.
Package facts
| License | BSD-3-Clause permissive |
| Python support | Supports the current Python release >=3.10 |
| Install friction | Low. Pure-Python wheel |
| Runtime dependencies | 2 packagesscrapyplaywright |
| Maintenance | Actively maintained 35 days since the last release |
| Last repo commit | |
| First released | |
| Downloads | 1,397,896 / month, #3,957 on PyPI 30-day window, as of 2026-08-14 |
| Known vulnerabilities | None known OSV.dev, checked 2026-08-14 |
| Classifiers | Development Status :: 4 - BetaFramework :: ScrapyIntended Audience :: DevelopersProgramming Language :: PythonProgramming Language :: Python :: 3.10Programming Language :: Python :: 3.11Programming Language :: Python :: 3.12Programming Language :: Python :: 3.13Programming Language :: Python :: 3.14Topic :: Internet :: WWW/HTTPTopic :: Software Development :: Libraries :: Application FrameworksTopic :: Software Development :: Libraries :: Python Modules |
Evidence: scrapy_playwright-0.0.48-py3-none-any.whl
Tags
Let your AI agent find packages like this
Example. Real query, live index.
You found this page by searching. An agent finds it by wishing: SkillFed indexes 14,416 PyPI packages by what they can do, searchable in plain language.
wish › “playwright scrapy integration”
- scrapy-playwrightIntegrates Playwright browser automation into Scrapy's download…
- scrapydoProvides a blocking API to run Scrapy spiders synchronously, wrapping…
- scrapy-zyte-apiScrapy plugin that integrates Zyte API for web scraping, allowing you…
Give your agent the search over MCP, or paste the wish link into any chat.
More Python Modules packages
Converts domain names between Unicode and ASCII-compatible encoding (Punycode) according to IDNA 2008 and Unicode Technical Standard 46, with security validation and broader script coverage than the standard library.
Install it if you work with internationalized domain names, need to validate domains, or use HTTP clients that depend on it transitively.
Setuptools is a Python build backend and package management tool that handles building, distributing, and installing Python packages, including support for C/C++ extension modules.
PyYAML parses and emits YAML 1.1 data format, enabling serialization and deserialization of configuration files and Python objects to and from human-readable YAML text.
Pydantic validates Python data structures against type hints, coercing and checking input at runtime to ensure it matches a declared schema.
Provides reusable metadata objects for use with PEP-593 `typing.Annotated` to express common constraints like bounds, collection sizes, and predicates on types.
Install it if you use or build libraries that need to express type constraints in a standardized, inspectable way—or if you want to annotate your own types with…
Provides runtime tools to inspect and introspect Python type annotations, enabling programmatic examination of type hints at execution time.
See also playwright · rebrowser-playwright · pytest-playwright · pytest-playwright-asyncio · robotframework-browser · robocorp-browser · scrapydo · install-playwright · scrapy-impersonate · scrapy-zyte-api