trafilatura

What it is and what it does

Trafilatura is a Python library and command-line tool for discovering, downloading, and extracting text and metadata from web pages. It handles the full pipeline: crawling via sitemaps and feeds, downloading HTML, parsing and cleaning content, and exporting results in formats like JSON, CSV, Markdown, XML, and HTML. The core extraction logic balances precision (removing noise like headers, footers, navigation) against recall (preserving valid content), and it supports optional extraction of comments, links, images, and tables alongside main text and metadata.

The package is designed for both one-off extraction and bulk processing, with parallel handling of online URLs and offline HTML files. It integrates widely in academic and commercial projects (HuggingFace, IBM, Microsoft Research, Stanford, Allen Institute) and is actively maintained with regular updates. Runtime dependencies include lxml for parsing, courlan for URL management, htmldate for date extraction, and justext for text segmentation.

Use it for:

• Build a text corpus from news websites or blogs by crawling sitemaps and extracting article text and publication dates.
• Convert downloaded HTML files to clean JSON or CSV for data analysis, removing boilerplate and preserving only article content.
• Extract metadata (title, author, date, categories) from web pages for indexing or cataloging.
• Process large batches of HTML documents in parallel to prepare training data for NLP or machine learning models.
• Scrape and structure web content for research purposes while respecting politeness and deduplication rules.

Worth the install?

AI-flagged interpretation of the facts on this page — verify before relying

Trafilatura extracts main text, metadata, and structured content from web pages and HTML, converting raw HTML into clean, usable data in multiple output formats.

Yes. Trafilatura is actively maintained, widely adopted, has no known vulnerabilities, and offers low install friction. It solves a real problem (converting messy HTML to clean text and metadata) with a mature, well-documented API. Use it if you need reliable web content extraction for research, data pipelines, or corpus building.

Install

trafilatura on PyPI

pip install trafilatura

uv add trafilatura

poetry add trafilatura

Installing trafilatura

Before you install

Low install friction with a pure-Python wheel distribution. Active maintenance with a recent release (14 days old) and strong community engagement (6633 GitHub stars). Supports Python 3.10 through 3.14.

License in practice

Distributed under Apache 2.0 (permissive), allowing commercial and private use with minimal restrictions. Versions prior to 1.8.0 were GPLv3+, but current releases impose no copyleft obligations.

Quickstart

pip install trafilatura

import trafilatura
downloaded = trafilatura.fetch_url('https://example.com')
result = trafilatura.extract(downloaded)
print(result)

Requires Python 3.10 or later. lxml (a runtime dependency) requires compilation; pre-built wheels are available for most platforms.

Verify before relying

• Whether the package's performance benchmarks (ScrapingHub, Bevendorff et al. 2023) remain current for version 2.2.0.
• Specific memory or CPU constraints when processing very large HTML documents or high-volume crawls.
• Whether language detection and speed optimization add-ons are included or require separate installation.

Package facts

Evidence: trafilatura-2.2.0-py3-none-any.whl

Keywords: corpus, html2text, news-crawler, natural-language-processing, scraper, tei-xml, text-extraction, webscraping, web-scraping

Tags

More WWW/HTTP packages