Protego

Pure-Python robots.txt parser with support for modern conventions

protego Permissive license BSD-3-Clause Active ★ 91 v0.6.2 released 2026-06-25

Install

protego on PyPI

pip

pip install protego

uv

uv add protego

poetry

poetry add protego

Package facts

License	BSD-3-Clause (permissive)
Python support	supports the current Python release (>=3.10)
Install friction	low — pure-Python wheel
Runtime dependencies	none
Maintenance	actively maintained — 49 days since the last release
Last repo commit	2026-07-27
First released	2019-07-18
Popularity	one of the top 1,000 most-downloaded packages on PyPI (30-day window, as of 2026-08-13)
Known vulnerabilities	none known (OSV.dev, checked 2026-08-13)

Evidence: protego-0.6.2-py3-none-any.whl

Keywords: parser, rep, robots, robots.txt

Development Status :: 4 - BetaIntended Audience :: DevelopersOperating System :: OS IndependentProgramming Language :: PythonProgramming Language :: Python :: 3Programming Language :: Python :: 3.10Programming Language :: Python :: 3.11Programming Language :: Python :: 3.12Programming Language :: Python :: 3.13Programming Language :: Python :: 3.14Programming Language :: Python :: Implementation :: CPythonProgramming Language :: Python :: Implementation :: PyPyTopic :: Internet :: WWW/HTTPTopic :: Software Development :: Libraries :: Python Modules

About Protego

from the package's own PyPI description — quoted content, verbatim

======= Protego =======

.. image:: https://img.shields.io/pypi/pyversions/protego.svg :target: https://pypi.python.org/pypi/protego :alt: Supported Python Versions

.. image:: https://github.com/scrapy/protego/actions/workflows/tests-ubuntu.yml/badge.svg :target: https://github.com/scrapy/protego/actions/workflows/tests-ubuntu.yml :alt: CI

Protego is a pure-Python robots.txt parser with support for modern conventions.

Install

To install Protego, simply use pip:

.. code-block:: none

pip install protego

Usage

.. code-block:: pycon

>>> from protego import Protego >>> robotstxt = """ ... User-agent: * ... Disallow: / ... Allow: /about ... Allow: /account ... Disallow: /account/contact$ ... Disallow: /account/*/profile ... Crawl-delay: 4 ... Request-rate: 10/1m # 10 requests every 1 minute ... ... Sitemap: http://example.com/sitemap-index.xml ... Host: http://example.co.in ... """ >>> rp = Protego.parse(robotstxt) >>> rp.can_fetch("http://example.com/profiles", "mybot") False >>> rp.can_fetch("http://example.com/about", "mybot") True >>>...

Read as markdown · JSON record · Source repository · Homepage

AI interpretation — verify before relying

AI-generated interpretation of the package facts above; every digit, version, license, or vulnerability id it cites is grounded in the facts already shown on this page

Protego is a pure-Python robots.txt parser that interprets modern conventions including wildcards, length-based precedence, crawl delays, request rates, and sitemaps to determine whether a user agent can fetch a given URL.

Low install friction; pure Python wheel with no runtime dependencies. Actively maintained with a release 49 days ago and recent commits; supports Python 3.10–3.14 on both CPython and PyPy.

BSD-3-Clause permissive license allows commercial and private use with minimal restrictions, requiring only preservation of copyright and license notices.

Usage

pip install protego

from protego import Protego
robotstxt = "User-agent: *\nDisallow: /\nAllow: /about"
rp = Protego.parse(robotstxt)
rp.can_fetch("http://example.com/about", "mybot")  # True

Requires Python 3.10 or later.

Verdict: Protego is a well-maintained, dependency-free robots.txt parser following Google's modern specification. No known vulnerabilities, permissive license, and active development make it a solid choice for web scraping and crawling compliance.

Needs verification

Whether the package is used or recommended by the Scrapy project itself beyond repository hosting.
Performance characteristics compared to alternatives in real-world scenarios beyond the benchmark cited in the description.

robots.txt parserweb scraping robots.txtuser agent fetch permissioncrawl delay extractionsitemap discoveryrobots exclusion rulesurl crawlability check