Protego
Pure-Python robots.txt parser with support for modern conventions
Decision gist · record as of 2026-08-14
Yes. Protego is a lightweight, actively maintained parser with no dependencies, permissive licensing, and support for modern robots.txt conventions. It's appropriate for any Python project that needs to respect or validate robots.txt rules. No known security vulnerabilities.AI-flagged interpretation of the facts on this page — verify before relying
Before you install
- Requires Python 3.10 or later.
- Low install friction with no runtime dependencies.
- Actively maintained as of June 2026, with recent commits and steady development.
License · maintenance · safety
BSD-3-Clause (permissive) — BSD-3-Clause permissive license allows commercial and private use with minimal restrictions.
last release 2026-06-25 (50 days) · last repo commit 2026-07-27 · 91 stars
0 known vulnerabilities (OSV.dev, 2026-08-14) · 23,559,448 downloads/mo, #938 on PyPI
Alternatives
Verify before relying
pip install protego
from protego import Protego
robotstxt = "User-agent: *\nDisallow: /\nAllow: /about"
rp = Protego.parse(robotstxt)
print(rp.can_fetch("http://example.com/about", "mybot"))- Whether the package handles all edge cases in real-world robots.txt files beyond the documented examples.
- Performance characteristics when parsing very large robots.txt files.
- Compatibility with non-standard or malformed robots.txt syntax.
What it is and what it does
Protego is a pure-Python robots.txt parser that interprets the rules in a robots.txt file and answers whether a specific URL can be fetched by a given user agent. It implements Google's modern robots.txt specification, including support for wildcard patterns and length-based precedence rules that determine which rule applies when multiple rules could match a URL. The parser also extracts metadata like crawl delays, request rates, sitemaps, and preferred hosts from the robots.txt file.
The package has no external runtime dependencies, making it lightweight and easy to integrate into web scrapers, crawlers, or any tool that needs to respect robots.txt rules. It's actively maintained and supports modern Python versions (3.10 through 3.14), with both CPython and PyPy implementations available.
Use it for
- Check if a web scraper or crawler can fetch a URL before making the request, respecting site crawl policies.
- Extract crawl delay and request rate limits from robots.txt to throttle requests appropriately.
- Retrieve sitemap URLs and preferred host information from a robots.txt file for indexing workflows.
- Validate compliance with robots.txt rules in automated testing of web crawling applications.
- Parse robots.txt from multiple domains to enforce uniform crawling behavior across a distributed crawler.
Worth the install?
AI-flagged interpretation of the facts on this page. Verify before relying on it.
Yes.
Protego is a lightweight, actively maintained parser with no dependencies, permissive licensing, and support for modern robots.txt conventions. It's appropriate for any Python project that needs to respect or validate robots.txt rules. No known security vulnerabilities.
Install
protego on PyPI
Before you install
Low install friction with no runtime dependencies. Actively maintained as of June 2026, with recent commits and steady development.
Requires Python 3.10 or later.
License in practice
BSD-3-Clause permissive license allows commercial and private use with minimal restrictions.
Quickstart
pip install protego
from protego import Protego
robotstxt = "User-agent: *\nDisallow: /\nAllow: /about"
rp = Protego.parse(robotstxt)
print(rp.can_fetch("http://example.com/about", "mybot"))Verify before relying
- Whether the package handles all edge cases in real-world robots.txt files beyond the documented examples.
- Performance characteristics when parsing very large robots.txt files.
- Compatibility with non-standard or malformed robots.txt syntax.
Package facts
| License | BSD-3-Clause permissive |
| Python support | Supports the current Python release >=3.10 |
| Install friction | Low. Pure-Python wheel |
| Runtime dependencies | None |
| Maintenance | Actively maintained 50 days since the last release |
| Last repo commit | |
| First released | |
| Downloads | 23,559,448 / month, #938 on PyPI 30-day window, as of 2026-08-14 |
| Known vulnerabilities | None known OSV.dev, checked 2026-08-14 |
| Classifiers | Development Status :: 4 - BetaIntended Audience :: DevelopersOperating System :: OS IndependentProgramming Language :: PythonProgramming Language :: Python :: 3Programming Language :: Python :: 3.10Programming Language :: Python :: 3.11Programming Language :: Python :: 3.12Programming Language :: Python :: 3.13Programming Language :: Python :: 3.14Programming Language :: Python :: Implementation :: CPythonProgramming Language :: Python :: Implementation :: PyPyTopic :: Internet :: WWW/HTTPTopic :: Software Development :: Libraries :: Python Modules |
Evidence: protego-0.6.2-py3-none-any.whl
Tags
Let your AI agent find packages like this
Example. Real query, live index.
You found this page by searching. An agent finds it by wishing: SkillFed indexes 14,416 PyPI packages by what they can do, searchable in plain language.
wish › “robots.txt parser”
- ProtegoProtego parses robots.txt files and determines whether a given URL…
- robotspyParses robots.txt files according to RFC 9309 and determines whether…
- ultimate-sitemap-parserParses and crawls sitemaps in multiple formats (XML, RSS, Atom, plain…
Give your agent the search over MCP, or paste the wish link into any chat.
More Python Modules packages
Converts domain names between Unicode and ASCII-compatible encoding (Punycode) according to IDNA 2008 and Unicode Technical Standard 46, with security validation and broader script coverage than the standard library.
Install it if you work with internationalized domain names, need to validate domains, or use HTTP clients that depend on it transitively.
Setuptools is a Python build backend and package management tool that handles building, distributing, and installing Python packages, including support for C/C++ extension modules.
PyYAML parses and emits YAML 1.1 data format, enabling serialization and deserialization of configuration files and Python objects to and from human-readable YAML text.
Pydantic validates Python data structures against type hints, coercing and checking input at runtime to ensure it matches a declared schema.
Provides reusable metadata objects for use with PEP-593 `typing.Annotated` to express common constraints like bounds, collection sizes, and predicates on types.
Install it if you use or build libraries that need to express type constraints in a standardized, inspectable way—or if you want to annotate your own types with…
Provides runtime tools to inspect and introspect Python type annotations, enabling programmatic examination of type hints at execution time.
See also robotspy · django-robots · ultimate-sitemap-parser · gitignorant · ua-parser · rfc3986-validator · user-agents · courlan · resolve-robotics-uri-py · tavily-cli