{"categories":[{"label":"WWW/HTTP","url":"https://skillfed.io/packages/category/internet-www-http/4"}],"enrichment":{"capability":"Parses robots.txt files according to RFC 9309 and determines whether a given user agent can fetch a specific path, with both a Python API and a command-line tool.","skillfed_tags":["web-crawling","compliance","cli-tool"],"use_cases":["Check if a web scraper or bot is permitted to crawl a specific URL path before making requests.","Validate robots.txt files programmatically as part of a web crawler or link checker project.","Build a command-line utility to quickly test whether a user agent can access a given path on a remote site.","Integrate RFC 9309-compliant robots.txt parsing into a Python application without external dependencies.","Migrate from a standard library implementation to a more spec-compliant parser using the compatible API."],"what_it_does":"robotspy is a parser for robots.txt files that implements the Robots Exclusion Protocol as defined in RFC 9309. It provides a primary class, RobotsParser, for parsing and querying robots.txt files, plus a compatibility facade (RobotFileParser) that mimics a standard library API. The package includes both a Python module for programmatic use and a command-line tool that can be invoked directly or via pipx for system-wide installation.\n\nThe parser determines whether a given user agent is allowed to fetch a specific path according to the rules in a robots.txt file. It supports the sitemaps directive and aims to follow RFC 9309 specs rather than non-standard extensions like request-rate or crawl-delay. It passes the same test suite as Google's Robots.txt Parser, except for Google-specific behaviors. The package has no external runtime dependencies, making it lightweight to integrate into web crawlers, link checkers, and compliance tools.","worth_installing":"Yes. The package has low install friction, no runtime dependencies, active maintenance, MIT licensing, and zero known vulnerabilities. It fills a clear need for RFC 9309-compliant robots.txt parsing with both library and CLI interfaces. Install it if you need to parse robots.txt files or check crawler permissions in a Python project."},"id":"robotspy","links":{"html":"https://skillfed.io/packages/robotspy","md":"https://skillfed.io/packages/robotspy.md","pypi":"https://pypi.org/project/robotspy/"},"maintenance":{"status":"active"},"meta":{"latest_release":"2026-03-01","license_spdx":null,"license_treatment":"permissive","name":"robotspy","python_support":"unspecified","summary":"Robots Exclusion Protocol File Parser"},"popularity":{"monthly_downloads":442529,"position":6636,"tier":"top_15000"},"security":{"n_vulnerabilities":0},"version":"0.13.0"}