---
id: protego
version: "0.6.2"
license: BSD-3-Clause
license_treatment: permissive
maintenance: active
---

# Protego — Pure-Python robots.txt parser with support for modern conventions

License: permissive · Maintenance: active · Popularity: top 1,000 on PyPI

## Install

pip install protego
uv add protego
poetry add protego

## Description

=======
Protego
=======

.. image:: https://img.shields.io/pypi/pyversions/protego.svg
   :target: https://pypi.python.org/pypi/protego
   :alt: Supported Python Versions

.. image:: https://github.com/scrapy/protego/actions/workflows/tests-ubuntu.yml/badge.svg
   :target: https://github.com/scrapy/protego/actions/workflows/tests-ubuntu.yml
   :alt: CI

Protego is a pure-Python ``robots.txt`` parser with support for modern
conventions.


Install
=======

To install Protego, simply use pip:

.. code-block:: none

    pip install protego


Usage
=====

.. code-block:: pycon

   >>> from protego import Protego
   >>> robotstxt = """
   ... User-agent: *
   ... Disallow: /
   ... Allow: /about
   ... Allow: /account
   ... Disallow: /account/contact$
   ... Disallow: /account/*/profile
   ... Crawl-delay: 4
   ... Request-rate: 10/1m                 # 10 requests every 1 minute
   ...
   ... Sitemap: http://example.com/sitemap-index.xml
   ... Host: http://example.co.in
   ... """
   >>> rp = Protego.parse(robotstxt)
   >>> rp.can_fetch("http://example.com/profiles", "mybot")
   False
   >>> rp.can_fetch("http://example.com/about", "mybot")
   True
   >>>...

## AI interpretation — verify before relying

Protego is a pure-Python robots.txt parser that interprets modern conventions including wildcards, length-based precedence, crawl delays, request rates, and sitemaps to determine whether a user agent can fetch a given URL.

Verdict: Protego is a well-maintained, dependency-free robots.txt parser following Google's modern specification. No known vulnerabilities, permissive license, and active development make it a solid choice for web scraping and crawling compliance.

[View on SkillFed](https://skillfed.io/packages/protego) · [View on PyPI](https://pypi.org/project/protego/)