{"categories":[{"label":"WWW/HTTP","url":"https://skillfed.io/packages/category/internet-www-http"},{"label":"HTML","url":"https://skillfed.io/packages/category/text-processing-markup-html"},{"label":"Filters","url":"https://skillfed.io/packages/category/text-processing-filters"},{"label":"Pre-processors","url":"https://skillfed.io/packages/category/software-development-pre-processors"}],"enrichment":{"capability":"jusText removes boilerplate content (navigation, headers, footers) from HTML pages while preserving text with full sentences, useful for extracting clean content for linguistic analysis and web corpora.","skillfed_tags":["content-extraction","web-scraping","nlp-preprocessing"],"use_cases":["Build web corpora for natural language processing by extracting clean text from crawled HTML pages","Preprocess web content before feeding it into linguistic analysis or text processing pipelines","Automate extraction of article text from news sites or blogs while discarding navigation and ads","Clean HTML snapshots for archival or readability analysis in web preservation workflows","Remove noise from web-scraped training data for text classification or language models"],"what_it_does":"jusText is a heuristic-based tool for extracting main content from HTML pages by filtering out boilerplate elements like navigation, headers, and footers. It operates by analyzing text density and sentence structure, making it particularly suited for building linguistic corpora and web-scraped datasets where clean, sentence-bearing text is needed.\n\nThe package provides both a command-line interface (via `python -m justext`) and a Python API. It depends on lxml for HTML parsing and includes language-specific stoplists to improve detection accuracy. The core algorithm preserves paragraphs containing full sentences while marking others as boilerplate, allowing you to filter results programmatically.","worth_installing":"Yes, with conditions. The package is stable (Production/Stable classifier), permissively licensed, and has low install friction. However, maintenance is aging (535 days since last release), so it is best suited for established use cases where the algorithm is known to work well for your content type. Verify that lxml compiles on your target platform and that boilerplate detection accuracy meets your needs before production use."},"id":"justext","links":{"html":"https://skillfed.io/packages/justext","md":"https://skillfed.io/packages/justext.md","pypi":"https://pypi.org/project/justext/"},"maintenance":{"status":"aging"},"meta":{"latest_release":"2025-02-25","license_spdx":null,"license_treatment":"permissive","name":"jusText","python_support":"unspecified","summary":"Heuristic based boilerplate removal tool"},"popularity":{"monthly_downloads":11719151,"position":1368,"tier":"top_5000"},"security":{"n_vulnerabilities":0},"version":"3.0.2"}