{"categories":[{"label":"Python Modules","url":"https://skillfed.io/packages/category/software-development-libraries-python-modules/7"},{"label":"WWW/HTTP","url":"https://skillfed.io/packages/category/internet-www-http/2"},{"label":"Application Frameworks","url":"https://skillfed.io/packages/category/software-development-libraries-application-frameworks/2"}],"enrichment":{"capability":"Integrates Playwright browser automation into Scrapy's download pipeline, enabling JavaScript-heavy pages to be scraped while maintaining Scrapy's standard request scheduling and processing workflow.","skillfed_tags":["web-scraping","browser-automation","javascript-rendering"],"use_cases":["Scrape single-page applications (SPAs) and client-rendered content that requires JavaScript execution to populate the DOM.","Handle pages with dynamic loading, infinite scroll, or AJAX-driven content that plain HTTP requests cannot capture.","Test or monitor web applications that depend on browser-specific behavior, user-agent matching, or JavaScript-based authentication.","Integrate browser automation into large-scale scraping pipelines without abandoning Scrapy's architecture and concurrency model.","Connect to remote browser instances (via CDP or WebSocket) for distributed or containerized scraping workflows."],"what_it_does":"scrapy-playwright is a Scrapy download handler that replaces the default HTTP/HTTPS handler to route requests through Playwright, a headless browser automation library. It lets you scrape JavaScript-rendered pages while keeping Scrapy's request scheduling, middleware, and item pipeline intact\u2014requests marked with `meta={\"playwright\": True}` run through the browser, others use the standard handler.\n\nThe package requires Scrapy \u22652.7 (for asyncio support) and Python \u22653.10. After pip install, you must run `playwright install` to download browser binaries. Configuration happens through Scrapy settings: you register the handler in `DOWNLOAD_HANDLERS`, set the Twisted asyncio reactor, and optionally tune browser launch options, browser type (chromium/firefox/webkit), or connect to a remote browser via Chrome DevTools Protocol or WebSocket.","worth_installing":"Yes, if you need to scrape JavaScript-heavy sites within a Scrapy project. The integration is clean, maintenance is active, and the license is permissive. Install friction is low (pure Python, two deps), but you must meet the version floor (Python \u22653.10, Scrapy \u22652.7) and run `playwright install` for browser binaries. No known vulnerabilities. Not worth it if your target sites are server-rendered or if you prefer a standalone browser automation tool over Scrapy integration."},"id":"scrapy-playwright","links":{"html":"https://skillfed.io/packages/scrapy-playwright","md":"https://skillfed.io/packages/scrapy-playwright.md","pypi":"https://pypi.org/project/scrapy-playwright/"},"maintenance":{"status":"active"},"meta":{"latest_release":"2026-07-10","license_spdx":"BSD-3-Clause","license_treatment":"permissive","name":"scrapy-playwright","python_support":"supports_current","summary":"Playwright integration for Scrapy"},"popularity":{"monthly_downloads":1397896,"position":3957,"tier":"top_5000"},"security":{"n_vulnerabilities":0},"version":"0.0.48"}