requests-html

What it is and what it does

requests-html is a web scraping and HTML parsing library that wraps the requests library with parsing capabilities. It lets you fetch web pages and extract data using CSS selectors (jQuery-style), XPath, or direct link enumeration. The library includes mocked user-agent headers, automatic redirect following, connection pooling, and cookie persistence—all the conveniences of requests plus parsing.

The package supports both synchronous and asynchronous workflows, and includes JavaScript rendering via pyppeteer for pages that require client-side execution. It's designed for developers who want to scrape or parse HTML without learning a heavy framework, though its dormant maintenance status (last release 2019-02-17) means it receives no active updates or security patches.

Use it for:

• Extract all links from a web page for crawling or analysis without manually parsing HTML.
• Scrape structured data (product names, prices, metadata) from websites using CSS or XPath selectors.
• Render and scrape JavaScript-heavy pages that require a browser engine to populate content.
• Build async scrapers that fetch multiple pages concurrently to speed up data collection.
• Parse HTML responses in a requests-based workflow without switching to a separate parsing library.

Worth the install?

AI-flagged interpretation of the facts on this page — verify before relying

Parses and scrapes HTML from web pages using a simple, intuitive API built on top of requests, with support for CSS selectors, XPath, JavaScript rendering, and async operations.

Yes, if you need simple HTML parsing and scraping for a one-off project or legacy codebase already using requests. No, if you're starting a new scraping project—the dormant maintenance (last release 2019, no active development) means you won't get bug fixes, security updates, or compatibility improvements for modern websites. Consider actively maintained alternatives like BeautifulSoup with requests, or Scrapy for larger projects.

Install

requests-html on PyPI

pip install requests-html

uv add requests-html

poetry add requests-html

Installing requests-html

Before you install

Low install friction with a pure-Python wheel. However, the package is dormant—last release was 2019-02-17 and the last commit 2024-06-19, so it receives no active maintenance. The pyppeteer dependency for JavaScript support adds a transitive browser automation layer that may require system dependencies.

License in practice

MIT license (permissive) means you can use, modify, and distribute this package freely in commercial and private projects with minimal restrictions.

Quickstart

from requests_html import HTMLSession

session = HTMLSession()
r = session.get('https://example.com')
links = r.html.links

pyppeteer (for JavaScript rendering) may require a Chromium binary; verify system dependencies before relying on JavaScript support.

Verify before relying

• Whether pyppeteer's Chromium dependency is automatically provisioned or requires manual setup on all platforms.
• Current compatibility with modern websites that use heavy JavaScript frameworks or anti-scraping measures.
• Whether dormant status means security patches or bug fixes are unlikely if issues arise.

Package facts

Evidence: requests_html-0.10.0-py3-none-any.whl

Tags

More WWW/HTTP packages