---
id: w3lib
version: "2.4.1"
license: BSD-3-Clause
license_treatment: permissive
maintenance: active
---

# w3lib — Library of web-related functions

License: permissive · Maintenance: active · Downloads: 9.3M/mo

## What it is and what it does

w3lib is a utility library for web-related tasks commonly needed in web scraping, HTTP clients, and HTML processing. It provides functions to strip HTML tags and comments, extract base URLs from HTML snippets, translate HTML entities, convert raw HTTP headers to dictionaries, construct HTTP authentication headers, normalize URLs according to browser rules, and parse query parameters from URLs.

The library has no runtime dependencies and installs cleanly as a pure Python wheel. It is actively maintained, supports current Python versions (3.10–3.14), and runs on both CPython and PyPy. With monthly downloads in the millions and stable status, it is a widely-used component in web automation and data extraction workflows.

Use it for:

- Strip HTML tags and comments from web page snippets before text analysis or storage
- Normalize and sanitize URLs in web scrapers to avoid duplicate requests or broken links
- Parse and manipulate HTTP headers when building custom HTTP clients or middleware
- Extract base URLs from HTML documents to resolve relative links correctly
- Construct HTTP Basic Authentication headers for requests to protected resources

## Worth the install?

AI-flagged interpretation of the facts above — verify before relying.

w3lib provides utility functions for common web tasks: parsing and manipulating HTML, extracting URLs and their components, handling HTTP headers and authentication, and normalizing URLs according to browser standards.

Yes. w3lib is a mature, zero-dependency utility library with active maintenance, no known vulnerabilities, permissive licensing, and broad Python version support. Install it if you need reliable, battle-tested helpers for URL normalization, HTML parsing, or HTTP header manipulation in web scraping or HTTP client code.

## Install

pip install w3lib
uv add w3lib
poetry add w3lib

## Installing w3lib

Before you install: Low friction installation with no runtime dependencies. Active maintenance with recent releases; last commit 2026-08-14 and latest release 2026-03-20. Supports modern Python versions (3.10 through 3.14) on both CPython and PyPy.

License in practice: BSD-3-Clause permissive license allows use in commercial and open-source projects with minimal restrictions—include a copy of the license and attribution.

Quickstart:

pip install w3lib

from w3lib.html import remove_tags, extract_base_url
from w3lib.url import sanitize_url

html = '<p>Hello</p>'
text = remove_tags(html)
url = sanitize_url('http://example.com/path')

Requires Python 3.10 or later.

Verify before relying:

- Whether the package handles HTML5 entity decoding or only a subset of HTML entities
- Performance characteristics when processing large HTML documents or many URLs
- Whether URL sanitization matches all major browser behaviors or a subset
- How long the package has been in active use and maintenance history

## Package facts

- License: BSD-3-Clause (permissive)
- Python support: supports_current
- Install friction: low
- Maintenance: active
- Downloads: 9.3M/month (top 5,000 on PyPI)
- Known vulnerabilities: none known

## Tags

html parsing utilities, url normalization, http header handling, web scraping helpers, html tag removal, url extraction, web-utilities, html-parsing, url-handling

[View on SkillFed](https://skillfed.io/packages/w3lib) · [View on PyPI](https://pypi.org/project/w3lib/)