scrapy-impersonate
Scrapy download handler that can impersonate browser fingerprints
Decision gist · record as of 2026-08-14
Yes, if you need Scrapy-based scraping against sites with fingerprint-based bot detection. The package is actively maintained, has low install friction, carries a permissive MIT license, and integrates cleanly into Scrapy's architecture. No security vulnerabilities are known. The main gotcha is the requirement for Python 3.10+ and explicit asyncio reactor configuration—verify that your Scrapy version and environment support this before committing.AI-flagged interpretation of the facts on this page — verify before relying
Before you install
- Requires Python 3.10 or later and the asyncio-based Twisted reactor to be explicitly configured in Scrapy settings.
- Low friction: pure Python wheel with only two runtime dependencies (curl-cffi and scrapy).
- Actively maintained with a recent release (88 days ago) and steady repository activity.
License · maintenance · safety
MIT (permissive) — MIT license is permissive; you can use, modify, and distribute this package freely in commercial or private projects with minimal restrictions.
last release 2026-05-18 (88 days) · last repo commit 2026-05-18 · 239 stars
0 known vulnerabilities (OSV.dev, 2026-08-14) · 132,327 downloads/mo, #11,556 on PyPI
Alternatives
Verify before relying
pip install scrapy-impersonate
# In settings.py or custom_settings:
DOWNLOAD_HANDLERS = {
"http": "scrapy_impersonate.ImpersonateDownloadHandler",
"https": "scrapy_impersonate.ImpersonateDownloadHandler",
}
TWISTED_REACTOR = "twisted.internet.asyncioreactor.AsyncioSelectorReactor"
USER_AGENT = ""
# In spider:
yield scrapy.Request(
url,
meta={"impersonate": "chrome133a"},
)- Whether curl-cffi's browser impersonation reliably defeats modern anti-bot systems beyond basic TLS/JA3 detection.
- Performance overhead compared to standard Scrapy HTTP requests.
- Compatibility with Scrapy middleware and request/response pipelines when using the custom download handler.
What it is and what it does
scrapy-impersonate is a Scrapy download handler that integrates curl-cffi to perform HTTP requests with spoofed browser fingerprints. Instead of Scrapy's default HTTP client, it routes requests through curl-cffi, which can mimic the TLS signatures and JA3 fingerprints of real browsers—Chrome, Firefox, Safari, and Edge across multiple versions and platforms. This allows web scrapers to bypass basic fingerprint-based bot detection that rejects requests from non-browser clients.
The package works by replacing Scrapy's standard download handlers and providing a middleware that can randomly rotate between supported browser profiles on a per-request basis. You configure it via Scrapy settings, then tag individual requests with an `impersonate` meta key to specify which browser to emulate. The asyncio-based Twisted reactor is required for proper async execution. It's designed for developers who need to scrape sites with fingerprint-based anti-bot protections but want to stay within the Scrapy framework rather than switching to a different HTTP client entirely.
Use it for
- Scrape sites that reject requests with non-browser TLS signatures by impersonating Chrome, Firefox, or Safari.
- Rotate browser fingerprints across requests to avoid detection patterns based on consistent client profiles.
- Test web applications' anti-bot defenses by simulating requests from different browser versions and platforms.
- Extract data from sites protected by basic JA3 fingerprint checks without rewriting scraper logic outside Scrapy.
- Combine browser impersonation with Scrapy's middleware and pipeline ecosystem for complex scraping workflows.
Worth the install?
AI-flagged interpretation of the facts on this page. Verify before relying on it.
Yes, if you need Scrapy-based scraping against sites with fingerprint-based bot detection.
The package is actively maintained, has low install friction, carries a permissive MIT license, and integrates cleanly into Scrapy's architecture. No security vulnerabilities are known. The main gotcha is the requirement for Python 3.10+ and explicit asyncio reactor configuration—verify that your Scrapy version and environment support this before committing.
Install
scrapy-impersonate on PyPI
Before you install
Low friction: pure Python wheel with only two runtime dependencies (curl-cffi and scrapy). Actively maintained with a recent release (88 days ago) and steady repository activity.
Requires Python 3.10 or later and the asyncio-based Twisted reactor to be explicitly configured in Scrapy settings.
License in practice
MIT license is permissive; you can use, modify, and distribute this package freely in commercial or private projects with minimal restrictions.
Quickstart
pip install scrapy-impersonate
# In settings.py or custom_settings:
DOWNLOAD_HANDLERS = {
"http": "scrapy_impersonate.ImpersonateDownloadHandler",
"https": "scrapy_impersonate.ImpersonateDownloadHandler",
}
TWISTED_REACTOR = "twisted.internet.asyncioreactor.AsyncioSelectorReactor"
USER_AGENT = ""
# In spider:
yield scrapy.Request(
url,
meta={"impersonate": "chrome133a"},
)Verify before relying
- Whether curl-cffi's browser impersonation reliably defeats modern anti-bot systems beyond basic TLS/JA3 detection.
- Performance overhead compared to standard Scrapy HTTP requests.
- Compatibility with Scrapy middleware and request/response pipelines when using the custom download handler.
Package facts
| License | MIT permissive |
| Python support | Supports the current Python release >=3.10 |
| Install friction | Low. Pure-Python wheel |
| Runtime dependencies | 2 packagescurl-cffiscrapy |
| Maintenance | Actively maintained 88 days since the last release |
| Last repo commit | |
| First released | |
| Downloads | 132,327 / month, #11,556 on PyPI 30-day window, as of 2026-08-14 |
| Known vulnerabilities | None known OSV.dev, checked 2026-08-14 |
| Classifiers | License :: OSI Approved :: MIT LicenseProgramming Language :: Python :: 3Programming Language :: Python :: 3.10Programming Language :: Python :: 3.11Programming Language :: Python :: 3.12Programming Language :: Python :: 3.13 |
Evidence: scrapy_impersonate-1.7.0-py3-none-any.whl
Tags
Let your AI agent find packages like this
Example. Real query, live index.
You found this page by searching. An agent finds it by wishing: SkillFed indexes 14,416 PyPI packages by what they can do, searchable in plain language.
wish › “scrapy browser impersonation”
- scrapy-impersonateA Scrapy download handler that replaces HTTP/HTTPS requests with…
- scrapy-playwrightIntegrates Playwright browser automation into Scrapy's download…
- curl-cfficurl_cffi provides Python bindings to libcurl with browser…
Give your agent the search over MCP, or paste the wish link into any chat.
More WWW/HTTP packages
urllib3 is an HTTP client library that provides thread-safe connection pooling, SSL/TLS verification, multipart file uploads, request retries, compression support, and proxy handling for Python applications.
Requests is a Python HTTP library that simplifies sending HTTP/1.1 requests with automatic handling of headers, authentication, cookies, and response parsing.
h11 is a pure-Python HTTP/1.1 protocol implementation that handles parsing and serializing HTTP messages without any built-in I/O, letting you integrate it with any network layer you choose.
HTTPX is a fully featured HTTP client library for Python that provides both sync and async APIs, with support for HTTP/1.1 and HTTP/2, plus an integrated command-line client.
Install it if you are building new projects or modernizing existing ones that rely on HTTP.
A minimal low-level HTTP client library that sends HTTP requests with thread-safe and task-safe connection pooling, supporting HTTP/1.1, HTTP/2, proxies, and both sync and async interfaces.
aiohttp is an async HTTP client and server framework built on asyncio, supporting both WebSockets and middleware-based routing for building concurrent web applications.
Install it if you need async HTTP client or server capabilities in asyncio-based applications.
See also curl-cffi · primp · rnet · scrapy-playwright · browserforge · browser-cookie3 · fake-headers · django-impersonate · wafer-py · tls-client