crawlerdetect
CrawlerDetect is a Python library designed to identify bots, crawlers, and spiders by analyzing their user agents.
Decision gist · record as of 2026-08-14
Yes. The package is actively maintained, has no dependencies, carries a permissive MIT license, and solves a common web application need with a straightforward API. The 1462 crawler patterns provide broad coverage. Install if you need reliable bot detection; the low friction and clean maintenance history make it a safe choice.AI-flagged interpretation of the facts on this page — verify before relying
Before you install
- Requires Python 3.10 or later (supports current versions only).
- Low friction: zero runtime dependencies, pure Python wheel.
- Active maintenance with a release 15 days ago and last commit on 2026-07-30.
License · maintenance · safety
MIT (permissive) — MIT license permits unrestricted use, modification, and distribution in commercial and private projects with minimal restrictions.
last release 2026-07-30 (15 days) · last repo commit 2026-07-30 · 45 stars
0 known vulnerabilities (OSV.dev, 2026-08-14) · 136,867 downloads/mo, #11,384 on PyPI
Alternatives
Verify before relying
from crawlerdetect import CrawlerDetect
crawler_detect = CrawlerDetect()
result = crawler_detect.isCrawler('Mozilla/5.0 (compatible; Sosospider/2.0; +http://help.soso.com/webspider.htm)')
matches = crawler_detect.getMatches()- How frequently the 1462 crawler patterns are updated relative to new bot releases.
- Performance characteristics when checking high-volume user agent strings.
- Whether header-based detection (Variant 3) improves accuracy over user-agent-only checks.
What it is and what it does
CrawlerDetect is a Python wrapper around a web crawler detection library that identifies bots, crawlers, and spiders by matching user agents and HTTP headers against a curated pattern database. It exposes two main methods: `isCrawler()` to check if a given user agent is a known crawler, and `getMatches()` to retrieve the name of any detected crawler. The library can accept a user agent string directly, initialize with a pre-set user agent, or analyze HTTP headers passed as a dictionary.
The package maintains zero runtime dependencies and is kept in sync with upstream crawler patterns. It's useful for web applications that need to distinguish legitimate traffic from automated crawlers—for analytics filtering, rate limiting, or access control. The detection is pattern-based rather than behavioral, so it relies on the completeness and currency of its signature database.
Use it for
- Filter crawler traffic from web analytics to isolate genuine user sessions.
- Implement rate limiting or access restrictions for detected bots in web applications.
- Identify and block malicious or unwanted crawlers at the request handler level.
- Log and monitor which crawlers are accessing your site and how frequently.
- Serve different content or responses to crawlers versus human visitors.
Worth the install?
AI-flagged interpretation of the facts on this page. Verify before relying on it.
Yes.
The package is actively maintained, has no dependencies, carries a permissive MIT license, and solves a common web application need with a straightforward API. The 1462 crawler patterns provide broad coverage. Install if you need reliable bot detection; the low friction and clean maintenance history make it a safe choice.
Install
crawlerdetect on PyPI
Before you install
Low friction: zero runtime dependencies, pure Python wheel. Active maintenance with a release 15 days ago and last commit on 2026-07-30.
Requires Python 3.10 or later (supports current versions only).
License in practice
MIT license permits unrestricted use, modification, and distribution in commercial and private projects with minimal restrictions.
Quickstart
from crawlerdetect import CrawlerDetect
crawler_detect = CrawlerDetect()
result = crawler_detect.isCrawler('Mozilla/5.0 (compatible; Sosospider/2.0; +http://help.soso.com/webspider.htm)')
matches = crawler_detect.getMatches()Verify before relying
- How frequently the 1462 crawler patterns are updated relative to new bot releases.
- Performance characteristics when checking high-volume user agent strings.
- Whether header-based detection (Variant 3) improves accuracy over user-agent-only checks.
Package facts
| License | MIT permissive |
| Python support | Supports the current Python release <4,>=3.10 |
| Install friction | Low. Pure-Python wheel |
| Runtime dependencies | None |
| Maintenance | Actively maintained 15 days since the last release |
| Last repo commit | |
| First released | |
| Downloads | 136,867 / month, #11,384 on PyPI 30-day window, as of 2026-08-14 |
| Known vulnerabilities | None known OSV.dev, checked 2026-08-14 |
Evidence: crawlerdetect-0.4.2-py3-none-any.whl
Tags
Let your AI agent find packages like this
Example. Real query, live index.
You found this page by searching. An agent finds it by wishing: SkillFed indexes 14,416 PyPI packages by what they can do, searchable in plain language.
wish › “bot detection user agent”
- crawlerdetectIdentifies bots, crawlers, and spiders by analyzing user agents and…
- ua-parserParses user-agent strings to extract browser, operating system, and…
- user-agentsParses browser user agent strings to identify device type (mobile,…
Give your agent the search over MCP, or paste the wish link into any chat.
More WWW/HTTP packages
urllib3 is an HTTP client library that provides thread-safe connection pooling, SSL/TLS verification, multipart file uploads, request retries, compression support, and proxy handling for Python applications.
Requests is a Python HTTP library that simplifies sending HTTP/1.1 requests with automatic handling of headers, authentication, cookies, and response parsing.
h11 is a pure-Python HTTP/1.1 protocol implementation that handles parsing and serializing HTTP messages without any built-in I/O, letting you integrate it with any network layer you choose.
HTTPX is a fully featured HTTP client library for Python that provides both sync and async APIs, with support for HTTP/1.1 and HTTP/2, plus an integrated command-line client.
Install it if you are building new projects or modernizing existing ones that rely on HTTP.
A minimal low-level HTTP client library that sends HTTP requests with thread-safe and task-safe connection pooling, supporting HTTP/1.1, HTTP/2, proxies, and both sync and async interfaces.
aiohttp is an async HTTP client and server framework built on asyncio, supporting both WebSockets and middleware-based routing for building concurrent web applications.
Install it if you need async HTTP client or server capabilities in asyncio-based applications.
See also device-detector · httpagentparser · ua-parser · woothee · user-agents · icrawler · crawlee · user-agent · pymobiledetect · django-robots