scrapy-impersonate

What it is and what it does

scrapy-impersonate is a Scrapy download handler that integrates curl-cffi to perform HTTP requests with spoofed browser fingerprints. Instead of Scrapy's default HTTP client, it routes requests through curl-cffi, which can mimic the TLS signatures and JA3 fingerprints of real browsers—Chrome, Firefox, Safari, and Edge across multiple versions and platforms. This allows web scrapers to bypass basic fingerprint-based bot detection that rejects requests from non-browser clients.

The package works by replacing Scrapy's standard download handlers and providing a middleware that can randomly rotate between supported browser profiles on a per-request basis. You configure it via Scrapy settings, then tag individual requests with an `impersonate` meta key to specify which browser to emulate. The asyncio-based Twisted reactor is required for proper async execution. It's designed for developers who need to scrape sites with fingerprint-based anti-bot protections but want to stay within the Scrapy framework rather than switching to a different HTTP client entirely.

Use it for:

• Scrape sites that reject requests with non-browser TLS signatures by impersonating Chrome, Firefox, or Safari.
• Rotate browser fingerprints across requests to avoid detection patterns based on consistent client profiles.
• Test web applications' anti-bot defenses by simulating requests from different browser versions and platforms.
• Extract data from sites protected by basic JA3 fingerprint checks without rewriting scraper logic outside Scrapy.
• Combine browser impersonation with Scrapy's middleware and pipeline ecosystem for complex scraping workflows.

Worth the install?

AI-flagged interpretation of the facts on this page — verify before relying

A Scrapy download handler that replaces HTTP/HTTPS requests with curl-cffi to impersonate browser TLS signatures and JA3 fingerprints, allowing scraping of sites that detect and block non-browser clients.

Yes, if you need Scrapy-based scraping against sites with fingerprint-based bot detection. The package is actively maintained, has low install friction, carries a permissive MIT license, and integrates cleanly into Scrapy's architecture. No security vulnerabilities are known. The main gotcha is the requirement for Python 3.10+ and explicit asyncio reactor configuration—verify that your Scrapy version and environment support this before committing.

Install

scrapy-impersonate on PyPI

pip install scrapy-impersonate

uv add scrapy-impersonate

poetry add scrapy-impersonate

Installing scrapy-impersonate

Before you install

Low friction: pure Python wheel with only two runtime dependencies (curl-cffi and scrapy). Actively maintained with a recent release (88 days ago) and steady repository activity.

License in practice

MIT license is permissive; you can use, modify, and distribute this package freely in commercial or private projects with minimal restrictions.

Quickstart

pip install scrapy-impersonate

# In settings.py or custom_settings:
DOWNLOAD_HANDLERS = {
    "http": "scrapy_impersonate.ImpersonateDownloadHandler",
    "https": "scrapy_impersonate.ImpersonateDownloadHandler",
}
TWISTED_REACTOR = "twisted.internet.asyncioreactor.AsyncioSelectorReactor"
USER_AGENT = ""

# In spider:
yield scrapy.Request(
    url,
    meta={"impersonate": "chrome133a"},
)

Requires Python 3.10 or later and the asyncio-based Twisted reactor to be explicitly configured in Scrapy settings.

Verify before relying

• Whether curl-cffi's browser impersonation reliably defeats modern anti-bot systems beyond basic TLS/JA3 detection.
• Performance overhead compared to standard Scrapy HTTP requests.
• Compatibility with Scrapy middleware and request/response pipelines when using the custom download handler.

Package facts

Evidence: scrapy_impersonate-1.7.0-py3-none-any.whl

Tags

More WWW/HTTP packages