w3lib
Library of web-related functions
Decision gist · record as of 2026-08-14
Yes. w3lib is a mature, zero-dependency utility library with active maintenance, no known vulnerabilities, permissive licensing, and broad Python version support. Install it if you need reliable, battle-tested helpers for URL normalization, HTML parsing, or HTTP header manipulation in web scraping or HTTP client code.AI-flagged interpretation of the facts on this page — verify before relying
Before you install
- Requires Python 3.10 or later.
- Low friction installation with no runtime dependencies.
- Active maintenance with recent releases; last commit 2026-08-14 and latest release 2026-03-20.
License · maintenance · safety
BSD-3-Clause (permissive) — BSD-3-Clause permissive license allows use in commercial and open-source projects with minimal restrictions—include a copy of the license and attribution.
last release 2026-03-20 (147 days) · last repo commit 2026-08-14 · 418 stars
0 known vulnerabilities (OSV.dev, 2026-08-14) · 9,275,745 downloads/mo, #1,551 on PyPI
Alternatives
Verify before relying
pip install w3lib
from w3lib.html import remove_tags, extract_base_url
from w3lib.url import sanitize_url
html = '<p>Hello</p>'
text = remove_tags(html)
url = sanitize_url('http://example.com/path')- Whether the package handles HTML5 entity decoding or only a subset of HTML entities
- Performance characteristics when processing large HTML documents or many URLs
- Whether URL sanitization matches all major browser behaviors or a subset
- How long the package has been in active use and maintenance history
What it is and what it does
w3lib is a utility library for web-related tasks commonly needed in web scraping, HTTP clients, and HTML processing. It provides functions to strip HTML tags and comments, extract base URLs from HTML snippets, translate HTML entities, convert raw HTTP headers to dictionaries, construct HTTP authentication headers, normalize URLs according to browser rules, and parse query parameters from URLs.
The library has no runtime dependencies and installs cleanly as a pure Python wheel. It is actively maintained, supports current Python versions (3.10–3.14), and runs on both CPython and PyPy. With monthly downloads in the millions and stable status, it is a widely-used component in web automation and data extraction workflows.
Use it for
- Strip HTML tags and comments from web page snippets before text analysis or storage
- Normalize and sanitize URLs in web scrapers to avoid duplicate requests or broken links
- Parse and manipulate HTTP headers when building custom HTTP clients or middleware
- Extract base URLs from HTML documents to resolve relative links correctly
- Construct HTTP Basic Authentication headers for requests to protected resources
Worth the install?
AI-flagged interpretation of the facts on this page. Verify before relying on it.
Yes.
w3lib is a mature, zero-dependency utility library with active maintenance, no known vulnerabilities, permissive licensing, and broad Python version support. Install it if you need reliable, battle-tested helpers for URL normalization, HTML parsing, or HTTP header manipulation in web scraping or HTTP client code.
Install
w3lib on PyPI
Before you install
Low friction installation with no runtime dependencies. Active maintenance with recent releases; last commit 2026-08-14 and latest release 2026-03-20. Supports modern Python versions (3.10 through 3.14) on both CPython and PyPy.
Requires Python 3.10 or later.
License in practice
BSD-3-Clause permissive license allows use in commercial and open-source projects with minimal restrictions—include a copy of the license and attribution.
Quickstart
pip install w3lib
from w3lib.html import remove_tags, extract_base_url
from w3lib.url import sanitize_url
html = '<p>Hello</p>'
text = remove_tags(html)
url = sanitize_url('http://example.com/path')Verify before relying
- Whether the package handles HTML5 entity decoding or only a subset of HTML entities
- Performance characteristics when processing large HTML documents or many URLs
- Whether URL sanitization matches all major browser behaviors or a subset
- How long the package has been in active use and maintenance history
Package facts
| License | BSD-3-Clause permissive |
| Python support | Supports the current Python release >=3.10 |
| Install friction | Low. Pure-Python wheel |
| Runtime dependencies | None |
| Maintenance | Actively maintained 147 days since the last release |
| Last repo commit | |
| First released | |
| Downloads | 9,275,745 / month, #1,551 on PyPI 30-day window, as of 2026-08-14 |
| Known vulnerabilities | None known OSV.dev, checked 2026-08-14 |
| Classifiers | Development Status :: 5 - Production/StableIntended Audience :: DevelopersOperating System :: OS IndependentProgramming Language :: PythonProgramming Language :: Python :: 3Programming Language :: Python :: 3.10Programming Language :: Python :: 3.11Programming Language :: Python :: 3.12Programming Language :: Python :: 3.13Programming Language :: Python :: 3.14Programming Language :: Python :: Implementation :: CPythonProgramming Language :: Python :: Implementation :: PyPyTopic :: Internet :: WWW/HTTPTopic :: Software Development :: Libraries :: Python Modules |
Evidence: w3lib-2.4.1-py3-none-any.whl
Tags
Let your AI agent find packages like this
Example. Real query, live index.
You found this page by searching. An agent finds it by wishing: SkillFed indexes 14,416 PyPI packages by what they can do, searchable in plain language.
wish › “html parsing utilities”
- w3libw3lib provides utility functions for common web tasks: parsing and…
- bx-py-utilsA collection of utility functions covering anonymization, dictionary…
- bs4This package is a placeholder that redirects users to install…
Give your agent the search over MCP, or paste the wish link into any chat.
More Python Modules packages
Converts domain names between Unicode and ASCII-compatible encoding (Punycode) according to IDNA 2008 and Unicode Technical Standard 46, with security validation and broader script coverage than the standard library.
Install it if you work with internationalized domain names, need to validate domains, or use HTTP clients that depend on it transitively.
Setuptools is a Python build backend and package management tool that handles building, distributing, and installing Python packages, including support for C/C++ extension modules.
PyYAML parses and emits YAML 1.1 data format, enabling serialization and deserialization of configuration files and Python objects to and from human-readable YAML text.
Pydantic validates Python data structures against type hints, coercing and checking input at runtime to ensure it matches a declared schema.
Provides reusable metadata objects for use with PEP-593 `typing.Annotated` to express common constraints like bounds, collection sizes, and predicates on types.
Install it if you use or build libraries that need to express type constraints in a standardized, inspectable way—or if you want to annotate your own types with…
Provides runtime tools to inspect and introspect Python type annotations, enabling programmatic examination of type hints at execution time.
See also netutils · turbohtml · readable-content · URLObject · html-text · kitchen · url-normalize · date-guesser · chompjs · html5lib