---
id: robotspy
version: "0.13.0"
license: MIT
license_treatment: permissive
maintenance: active
---

# robotspy — Robots Exclusion Protocol File Parser

License: permissive · Maintenance: active · Downloads: 442.5K/mo

## What it is and what it does

robotspy is a parser for robots.txt files that implements the Robots Exclusion Protocol as defined in RFC 9309. It provides a primary class, RobotsParser, for parsing and querying robots.txt files, plus a compatibility facade (RobotFileParser) that mimics a standard library API. The package includes both a Python module for programmatic use and a command-line tool that can be invoked directly or via pipx for system-wide installation.

The parser determines whether a given user agent is allowed to fetch a specific path according to the rules in a robots.txt file. It supports the sitemaps directive and aims to follow RFC 9309 specs rather than non-standard extensions like request-rate or crawl-delay. It passes the same test suite as Google's Robots.txt Parser, except for Google-specific behaviors. The package has no external runtime dependencies, making it lightweight to integrate into web crawlers, link checkers, and compliance tools.

Use it for:

- Check if a web scraper or bot is permitted to crawl a specific URL path before making requests.
- Validate robots.txt files programmatically as part of a web crawler or link checker project.
- Build a command-line utility to quickly test whether a user agent can access a given path on a remote site.
- Integrate RFC 9309-compliant robots.txt parsing into a Python application without external dependencies.
- Migrate from a standard library implementation to a more spec-compliant parser using the compatible API.

## Worth the install?

AI-flagged interpretation of the facts above — verify before relying.

Parses robots.txt files according to RFC 9309 and determines whether a given user agent can fetch a specific path, with both a Python API and a command-line tool.

Yes. The package has low install friction, no runtime dependencies, active maintenance, MIT licensing, and zero known vulnerabilities. It fills a clear need for RFC 9309-compliant robots.txt parsing with both library and CLI interfaces. Install it if you need to parse robots.txt files or check crawler permissions in a Python project.

## Install

pip install robotspy
uv add robotspy
poetry add robotspy

## Installing robotspy

Before you install: Low install friction with no runtime dependencies. Actively maintained with a recent commit on 2026-06-19 and a release on 2026-03-01.

License in practice: MIT license permits unrestricted use, modification, and distribution with minimal restrictions.

Quickstart:

pip install robotspy

from robotspy import RobotsParser
parser = RobotsParser.from_uri('http://example.com/robots.txt')
can_fetch = parser.can_fetch('MyBot', '/path/to/page')
print(can_fetch)

Verify before relying:

- Whether the package handles all edge cases in RFC 9309 or if there are known deviations from the spec.
- Performance characteristics when parsing very large robots.txt files.
- Whether the RobotFileParser compatibility layer covers all use cases or has known limitations.

## Package facts

- License: MIT (permissive)
- Python support: unspecified
- Install friction: low
- Maintenance: active
- Downloads: 442.5K/month (top 15,000 on PyPI)
- Known vulnerabilities: none known

## Tags

robots.txt parser, robots exclusion protocol, check user agent access, web crawler permissions, RFC 9309 robots parser, robots file validator, web scraping compliance, web-crawling, compliance, cli-tool

[View on SkillFed](https://skillfed.io/packages/robotspy) · [View on PyPI](https://pypi.org/project/robotspy/)