---
id: crawlerdetect
version: "0.4.2"
license: MIT
license_treatment: permissive
maintenance: active
---

# crawlerdetect — CrawlerDetect is a Python library designed to identify bots, crawlers, and spiders by analyzing their user agents.

License: permissive · Maintenance: active · Downloads: 136.9K/mo

## What it is and what it does

CrawlerDetect is a Python wrapper around a web crawler detection library that identifies bots, crawlers, and spiders by matching user agents and HTTP headers against a curated pattern database. It exposes two main methods: `isCrawler()` to check if a given user agent is a known crawler, and `getMatches()` to retrieve the name of any detected crawler. The library can accept a user agent string directly, initialize with a pre-set user agent, or analyze HTTP headers passed as a dictionary.

The package maintains zero runtime dependencies and is kept in sync with upstream crawler patterns. It's useful for web applications that need to distinguish legitimate traffic from automated crawlers—for analytics filtering, rate limiting, or access control. The detection is pattern-based rather than behavioral, so it relies on the completeness and currency of its signature database.

Use it for:

- Filter crawler traffic from web analytics to isolate genuine user sessions.
- Implement rate limiting or access restrictions for detected bots in web applications.
- Identify and block malicious or unwanted crawlers at the request handler level.
- Log and monitor which crawlers are accessing your site and how frequently.
- Serve different content or responses to crawlers versus human visitors.

## Worth the install?

AI-flagged interpretation of the facts above — verify before relying.

Identifies bots, crawlers, and spiders by analyzing user agents and HTTP headers against a pattern database of 1462 crawler signatures.

Yes. The package is actively maintained, has no dependencies, carries a permissive MIT license, and solves a common web application need with a straightforward API. The 1462 crawler patterns provide broad coverage. Install if you need reliable bot detection; the low friction and clean maintenance history make it a safe choice.

## Install

pip install crawlerdetect
uv add crawlerdetect
poetry add crawlerdetect

## Installing crawlerdetect

Before you install: Low friction: zero runtime dependencies, pure Python wheel. Active maintenance with a release 15 days ago and last commit on 2026-07-30.

License in practice: MIT license permits unrestricted use, modification, and distribution in commercial and private projects with minimal restrictions.

Quickstart:

from crawlerdetect import CrawlerDetect
crawler_detect = CrawlerDetect()
result = crawler_detect.isCrawler('Mozilla/5.0 (compatible; Sosospider/2.0; +http://help.soso.com/webspider.htm)')
matches = crawler_detect.getMatches()

Requires Python 3.10 or later (supports current versions only).

Verify before relying:

- How frequently the 1462 crawler patterns are updated relative to new bot releases.
- Performance characteristics when checking high-volume user agent strings.
- Whether header-based detection (Variant 3) improves accuracy over user-agent-only checks.

## Package facts

- License: MIT (permissive)
- Python support: supports_current
- Install friction: low
- Maintenance: active
- Downloads: 136.9K/month (top 15,000 on PyPI)
- Known vulnerabilities: none known

## Tags

bot detection user agent, crawler detection library, spider identification, web crawler detection, user agent analysis, bot detection python, crawler pattern matching, bot-detection, user-agent-parsing

[View on SkillFed](https://skillfed.io/packages/crawlerdetect) · [View on PyPI](https://pypi.org/project/crawlerdetect/)