crawlerdetect

What it is and what it does

CrawlerDetect is a Python wrapper around a web crawler detection library that identifies bots, crawlers, and spiders by matching user agents and HTTP headers against a curated pattern database. It exposes two main methods: `isCrawler()` to check if a given user agent is a known crawler, and `getMatches()` to retrieve the name of any detected crawler. The library can accept a user agent string directly, initialize with a pre-set user agent, or analyze HTTP headers passed as a dictionary.

The package maintains zero runtime dependencies and is kept in sync with upstream crawler patterns. It's useful for web applications that need to distinguish legitimate traffic from automated crawlers—for analytics filtering, rate limiting, or access control. The detection is pattern-based rather than behavioral, so it relies on the completeness and currency of its signature database.

Use it for:

• Filter crawler traffic from web analytics to isolate genuine user sessions.
• Implement rate limiting or access restrictions for detected bots in web applications.
• Identify and block malicious or unwanted crawlers at the request handler level.
• Log and monitor which crawlers are accessing your site and how frequently.
• Serve different content or responses to crawlers versus human visitors.

Worth the install?

AI-flagged interpretation of the facts on this page — verify before relying

Identifies bots, crawlers, and spiders by analyzing user agents and HTTP headers against a pattern database of 1462 crawler signatures.

Yes. The package is actively maintained, has no dependencies, carries a permissive MIT license, and solves a common web application need with a straightforward API. The 1462 crawler patterns provide broad coverage. Install if you need reliable bot detection; the low friction and clean maintenance history make it a safe choice.

Install

crawlerdetect on PyPI

pip install crawlerdetect

uv add crawlerdetect

poetry add crawlerdetect

Installing crawlerdetect

Before you install

Low friction: zero runtime dependencies, pure Python wheel. Active maintenance with a release 15 days ago and last commit on 2026-07-30.

License in practice

MIT license permits unrestricted use, modification, and distribution in commercial and private projects with minimal restrictions.

Quickstart

from crawlerdetect import CrawlerDetect
crawler_detect = CrawlerDetect()
result = crawler_detect.isCrawler('Mozilla/5.0 (compatible; Sosospider/2.0; +http://help.soso.com/webspider.htm)')
matches = crawler_detect.getMatches()

Requires Python 3.10 or later (supports current versions only).

Verify before relying

• How frequently the 1462 crawler patterns are updated relative to new bot releases.
• Performance characteristics when checking high-volume user agent strings.
• Whether header-based detection (Variant 3) improves accuracy over user-agent-only checks.

Package facts

Evidence: crawlerdetect-0.4.2-py3-none-any.whl

Keywords: crawler, crawler detect, crawler detector, crawlerdetect, python crawler detect

Tags

More WWW/HTTP packages

Further reading