jusText

What it is and what it does

jusText is a heuristic-based tool for extracting main content from HTML pages by filtering out boilerplate elements like navigation, headers, and footers. It operates by analyzing text density and sentence structure, making it particularly suited for building linguistic corpora and web-scraped datasets where clean, sentence-bearing text is needed.

The package provides both a command-line interface (via `python -m justext`) and a Python API. It depends on lxml for HTML parsing and includes language-specific stoplists to improve detection accuracy. The core algorithm preserves paragraphs containing full sentences while marking others as boilerplate, allowing you to filter results programmatically.

Use it for:

• Build web corpora for natural language processing by extracting clean text from crawled HTML pages
• Preprocess web content before feeding it into linguistic analysis or text processing pipelines
• Automate extraction of article text from news sites or blogs while discarding navigation and ads
• Clean HTML snapshots for archival or readability analysis in web preservation workflows
• Remove noise from web-scraped training data for text classification or language models

Worth the install?

AI-flagged interpretation of the facts on this page — verify before relying

jusText removes boilerplate content (navigation, headers, footers) from HTML pages while preserving text with full sentences, useful for extracting clean content for linguistic analysis and web corpora.

Yes, with conditions. The package is stable (Production/Stable classifier), permissively licensed, and has low install friction. However, maintenance is aging (535 days since last release), so it is best suited for established use cases where the algorithm is known to work well for your content type. Verify that lxml compiles on your target platform and that boilerplate detection accuracy meets your needs before production use.

Install

justext on PyPI

pip install justext

uv add justext

poetry add justext

Installing jusText

Before you install

Low friction: pure Python wheel, only two runtime dependencies (lxml and backports.functools-lru-cache). Maintenance status is aging—last release 535 days ago, but repository is active (last commit 2025-02-25) with no archived flag.

License in practice

BSD 2-Clause License (permissive). No restrictions on commercial use, modification, or redistribution; standard permissive terms apply.

Quickstart

import justext

paragraphs = justext.justext(html_content, justext.get_stoplist("English"))
for paragraph in paragraphs:
    if not paragraph.is_boilerplate:
        print(paragraph.text)

lxml requires libxml2 and libxslt system libraries; may need compilation on some platforms.

Verify before relying

• Performance characteristics on very large HTML documents or high-throughput scenarios
• Accuracy of boilerplate detection across non-English languages and modern web layouts
• Whether backports.functools-lru-cache is still required on modern Python versions

Package facts

Evidence: justext-3.0.2-py2.py3-none-any.whl

Tags

More WWW/HTTP packages