---
id: justext
version: "3.0.2"
license: The BSD 2-Clause License
license_treatment: permissive
maintenance: aging
---

# jusText — Heuristic based boilerplate removal tool

License: permissive · Maintenance: aging · Downloads: 11.7M/mo

## What it is and what it does

jusText is a heuristic-based tool for extracting main content from HTML pages by filtering out boilerplate elements like navigation, headers, and footers. It operates by analyzing text density and sentence structure, making it particularly suited for building linguistic corpora and web-scraped datasets where clean, sentence-bearing text is needed.

The package provides both a command-line interface (via `python -m justext`) and a Python API. It depends on lxml for HTML parsing and includes language-specific stoplists to improve detection accuracy. The core algorithm preserves paragraphs containing full sentences while marking others as boilerplate, allowing you to filter results programmatically.

Use it for:

- Build web corpora for natural language processing by extracting clean text from crawled HTML pages
- Preprocess web content before feeding it into linguistic analysis or text processing pipelines
- Automate extraction of article text from news sites or blogs while discarding navigation and ads
- Clean HTML snapshots for archival or readability analysis in web preservation workflows
- Remove noise from web-scraped training data for text classification or language models

## Worth the install?

AI-flagged interpretation of the facts above — verify before relying.

jusText removes boilerplate content (navigation, headers, footers) from HTML pages while preserving text with full sentences, useful for extracting clean content for linguistic analysis and web corpora.

Yes, with conditions. The package is stable (Production/Stable classifier), permissively licensed, and has low install friction. However, maintenance is aging (535 days since last release), so it is best suited for established use cases where the algorithm is known to work well for your content type. Verify that lxml compiles on your target platform and that boilerplate detection accuracy meets your needs before production use.

## Install

pip install justext
uv add justext
poetry add justext

## Installing jusText

Before you install: Low friction: pure Python wheel, only two runtime dependencies (lxml and backports.functools-lru-cache). Maintenance status is aging—last release 535 days ago, but repository is active (last commit 2025-02-25) with no archived flag.

License in practice: BSD 2-Clause License (permissive). No restrictions on commercial use, modification, or redistribution; standard permissive terms apply.

Quickstart:

import justext

paragraphs = justext.justext(html_content, justext.get_stoplist("English"))
for paragraph in paragraphs:
    if not paragraph.is_boilerplate:
        print(paragraph.text)

lxml requires libxml2 and libxslt system libraries; may need compilation on some platforms.

Verify before relying:

- Performance characteristics on very large HTML documents or high-throughput scenarios
- Accuracy of boilerplate detection across non-English languages and modern web layouts
- Whether backports.functools-lru-cache is still required on modern Python versions

## Package facts

- License: The BSD 2-Clause License (permissive)
- Python support: unspecified
- Install friction: low
- Maintenance: aging
- Downloads: 11.7M/month (top 5,000 on PyPI)
- Known vulnerabilities: none known

## Tags

html boilerplate removal, extract main text from html, web content extraction, remove navigation headers footers, html text cleaning, linguistic corpus building, boilerplate stripping, content-extraction, web-scraping, nlp-preprocessing

[View on SkillFed](https://skillfed.io/packages/justext) · [View on PyPI](https://pypi.org/project/justext/)