robotspy
Robots Exclusion Protocol File Parser
Decision gist · record as of 2026-08-14
Yes. The package has low install friction, no runtime dependencies, active maintenance, MIT licensing, and zero known vulnerabilities. It fills a clear need for RFC 9309-compliant robots.txt parsing with both library and CLI interfaces. Install it if you need to parse robots.txt files or check crawler permissions in a Python project.AI-flagged interpretation of the facts on this page — verify before relying
Before you install
- Low install friction with no runtime dependencies.
- Actively maintained with a recent commit on 2026-06-19 and a release on 2026-03-01.
License · maintenance · safety
MIT (permissive) — MIT license permits unrestricted use, modification, and distribution with minimal restrictions.
last release 2026-03-01 (166 days) · last repo commit 2026-06-19 · 22 stars
0 known vulnerabilities (OSV.dev, 2026-08-14) · 442,529 downloads/mo, #6,636 on PyPI
Alternatives
Verify before relying
pip install robotspy
from robotspy import RobotsParser
parser = RobotsParser.from_uri('http://example.com/robots.txt')
can_fetch = parser.can_fetch('MyBot', '/path/to/page')
print(can_fetch)- Whether the package handles all edge cases in RFC 9309 or if there are known deviations from the spec.
- Performance characteristics when parsing very large robots.txt files.
- Whether the RobotFileParser compatibility layer covers all use cases or has known limitations.
What it is and what it does
robotspy is a parser for robots.txt files that implements the Robots Exclusion Protocol as defined in RFC 9309. It provides a primary class, RobotsParser, for parsing and querying robots.txt files, plus a compatibility facade (RobotFileParser) that mimics a standard library API. The package includes both a Python module for programmatic use and a command-line tool that can be invoked directly or via pipx for system-wide installation.
The parser determines whether a given user agent is allowed to fetch a specific path according to the rules in a robots.txt file. It supports the sitemaps directive and aims to follow RFC 9309 specs rather than non-standard extensions like request-rate or crawl-delay. It passes the same test suite as Google's Robots.txt Parser, except for Google-specific behaviors. The package has no external runtime dependencies, making it lightweight to integrate into web crawlers, link checkers, and compliance tools.
Use it for
- Check if a web scraper or bot is permitted to crawl a specific URL path before making requests.
- Validate robots.txt files programmatically as part of a web crawler or link checker project.
- Build a command-line utility to quickly test whether a user agent can access a given path on a remote site.
- Integrate RFC 9309-compliant robots.txt parsing into a Python application without external dependencies.
- Migrate from a standard library implementation to a more spec-compliant parser using the compatible API.
Worth the install?
AI-flagged interpretation of the facts on this page. Verify before relying on it.
Yes.
The package has low install friction, no runtime dependencies, active maintenance, MIT licensing, and zero known vulnerabilities. It fills a clear need for RFC 9309-compliant robots.txt parsing with both library and CLI interfaces. Install it if you need to parse robots.txt files or check crawler permissions in a Python project.
Install
robotspy on PyPI
Before you install
Low install friction with no runtime dependencies. Actively maintained with a recent commit on 2026-06-19 and a release on 2026-03-01.
License in practice
MIT license permits unrestricted use, modification, and distribution with minimal restrictions.
Quickstart
pip install robotspy
from robotspy import RobotsParser
parser = RobotsParser.from_uri('http://example.com/robots.txt')
can_fetch = parser.can_fetch('MyBot', '/path/to/page')
print(can_fetch)Verify before relying
- Whether the package handles all edge cases in RFC 9309 or if there are known deviations from the spec.
- Performance characteristics when parsing very large robots.txt files.
- Whether the RobotFileParser compatibility layer covers all use cases or has known limitations.
Package facts
| License | MIT permissive |
| Python support | Not specified |
| Install friction | Low. Pure-Python wheel |
| Runtime dependencies | None |
| Maintenance | Actively maintained 166 days since the last release |
| Last repo commit | |
| First released | |
| Downloads | 442,529 / month, #6,636 on PyPI 30-day window, as of 2026-08-14 |
| Known vulnerabilities | None known OSV.dev, checked 2026-08-14 |
| Classifiers | License :: OSI Approved :: MIT LicenseProgramming Language :: Python :: 3Programming Language :: Python :: 3.10Programming Language :: Python :: 3.11Programming Language :: Python :: 3.12Programming Language :: Python :: 3.13Programming Language :: Python :: 3.14 |
Evidence: robotspy-0.13.0-py3-none-any.whl
Tags
Let your AI agent find packages like this
Example. Real query, live index.
You found this page by searching. An agent finds it by wishing: SkillFed indexes 14,416 PyPI packages by what they can do, searchable in plain language.
wish › “check user agent access”
- robotspyParses robots.txt files according to RFC 9309 and determines whether…
- crawlerdetectIdentifies bots, crawlers, and spiders by analyzing user agents and…
- ProtegoProtego parses robots.txt files and determines whether a given URL…
Give your agent the search over MCP, or paste the wish link into any chat.
More WWW/HTTP packages
urllib3 is an HTTP client library that provides thread-safe connection pooling, SSL/TLS verification, multipart file uploads, request retries, compression support, and proxy handling for Python applications.
Requests is a Python HTTP library that simplifies sending HTTP/1.1 requests with automatic handling of headers, authentication, cookies, and response parsing.
h11 is a pure-Python HTTP/1.1 protocol implementation that handles parsing and serializing HTTP messages without any built-in I/O, letting you integrate it with any network layer you choose.
HTTPX is a fully featured HTTP client library for Python that provides both sync and async APIs, with support for HTTP/1.1 and HTTP/2, plus an integrated command-line client.
Install it if you are building new projects or modernizing existing ones that rely on HTTP.
A minimal low-level HTTP client library that sends HTTP requests with thread-safe and task-safe connection pooling, supporting HTTP/1.1, HTTP/2, proxies, and both sync and async interfaces.
aiohttp is an async HTTP client and server framework built on asyncio, supporting both WebSockets and middleware-based routing for building concurrent web applications.
Install it if you need async HTTP client or server capabilities in asyncio-based applications.
See also django-robots · Protego · ua-parser · woothee · yourdfpy · resolve-robotics-uri-py · uritools · ua-parser-rs · pylitterbot · ultimate-sitemap-parser