robotspy

What it is and what it does

robotspy is a parser for robots.txt files that implements the Robots Exclusion Protocol as defined in RFC 9309. It provides a primary class, RobotsParser, for parsing and querying robots.txt files, plus a compatibility facade (RobotFileParser) that mimics a standard library API. The package includes both a Python module for programmatic use and a command-line tool that can be invoked directly or via pipx for system-wide installation.

The parser determines whether a given user agent is allowed to fetch a specific path according to the rules in a robots.txt file. It supports the sitemaps directive and aims to follow RFC 9309 specs rather than non-standard extensions like request-rate or crawl-delay. It passes the same test suite as Google's Robots.txt Parser, except for Google-specific behaviors. The package has no external runtime dependencies, making it lightweight to integrate into web crawlers, link checkers, and compliance tools.

Use it for:

• Check if a web scraper or bot is permitted to crawl a specific URL path before making requests.
• Validate robots.txt files programmatically as part of a web crawler or link checker project.
• Build a command-line utility to quickly test whether a user agent can access a given path on a remote site.
• Integrate RFC 9309-compliant robots.txt parsing into a Python application without external dependencies.
• Migrate from a standard library implementation to a more spec-compliant parser using the compatible API.

Worth the install?

AI-flagged interpretation of the facts on this page — verify before relying

Parses robots.txt files according to RFC 9309 and determines whether a given user agent can fetch a specific path, with both a Python API and a command-line tool.

Yes. The package has low install friction, no runtime dependencies, active maintenance, MIT licensing, and zero known vulnerabilities. It fills a clear need for RFC 9309-compliant robots.txt parsing with both library and CLI interfaces. Install it if you need to parse robots.txt files or check crawler permissions in a Python project.

Install

robotspy on PyPI

pip install robotspy

uv add robotspy

poetry add robotspy

Installing robotspy

Before you install

Low install friction with no runtime dependencies. Actively maintained with a recent commit on 2026-06-19 and a release on 2026-03-01.

License in practice

MIT license permits unrestricted use, modification, and distribution with minimal restrictions.

Quickstart

pip install robotspy

from robotspy import RobotsParser
parser = RobotsParser.from_uri('http://example.com/robots.txt')
can_fetch = parser.can_fetch('MyBot', '/path/to/page')
print(can_fetch)

Verify before relying

• Whether the package handles all edge cases in RFC 9309 or if there are known deviations from the spec.
• Performance characteristics when parsing very large robots.txt files.
• Whether the RobotFileParser compatibility layer covers all use cases or has known limitations.

Package facts

Evidence: robotspy-0.13.0-py3-none-any.whl

Tags

More WWW/HTTP packages