clean-text

What it is and what it does

clean-text is a text preprocessing library that normalizes messy user-generated content from the web and social media. It fixes unicode encoding errors, removes or replaces unwanted patterns (URLs, emails, phone numbers, IP addresses, code snippets), handles transliteration to ASCII, and supports language-specific rules for English and German. The package wraps ftfy for unicode repair and optionally uses unidecode for transliteration, falling back to Python's built-in unicodedata when unidecode is unavailable.

The library offers both a simple functional API (clean() for single strings, clean_texts() for batch processing with optional multiprocessing) and a scikit-learn compatible transformer for integration into ML pipelines. You can preserve specific text patterns using regex-based exceptions to prevent them from being modified during cleaning, and customize replacement tokens for different content types.

Use it for:

• Normalize scraped web or social media text before feeding it into NLP models or search indexing.
• Batch-clean large text datasets in parallel using n_jobs to speed up preprocessing on multi-core systems.
• Standardize user input in web applications by removing malformed unicode, URLs, and email addresses while preserving legitimate content.
• Prepare text for machine learning pipelines via the scikit-learn CleanTransformer for reproducible preprocessing steps.
• Transliterate accented or non-ASCII characters to ASCII equivalents for systems that require ASCII-only text.

Worth the install?

AI-flagged interpretation of the facts on this page — verify before relying

Preprocesses and normalizes user-generated text by fixing unicode errors, removing or replacing URLs, emails, code snippets, and other patterns, with optional transliteration to ASCII and support for parallel batch processing.

Yes, if you need to clean messy user-generated text before NLP or indexing. The package is stable, permissively licensed, and has low install friction. The aging maintenance status (198 days since last release) is a minor concern but not a blocker for a mature text-cleaning utility. Consider it a solid choice for preprocessing pipelines, especially if you already use scikit-learn.

Install

clean-text on PyPI

pip install clean-text

uv add clean-text

poetry add clean-text

Installing clean-text

Before you install

Low friction install with two lightweight runtime dependencies (emoji and ftfy). Last release was 198 days ago; maintenance status is aging but the package remains functional for its core use case.

License in practice

Licensed under Apache-2.0 (permissive), allowing commercial and private use with minimal restrictions. No GPL obligation unless you explicitly install the optional unidecode extra.

Quickstart

pip install clean-text

from cleantext import clean

result = clean("Yóù àré right!", fix_unicode=True, to_ascii=True, lower=True)

Requires Python 3.9 or later. Optional unidecode dependency (installed via clean-text[gpl]) provides better transliteration but carries GPL licensing; without it, falls back to Python's built-in unicodedata with slightly different output.

Verify before relying

• Whether the aging maintenance status (198 days since last release) affects bug fixes or feature requests going forward.
• Real-world performance of parallel processing (n_jobs parameter) on typical workloads.
• Consistency and quality differences between transliteration with and without unidecode in production use.

Package facts

Evidence: clean_text-0.7.1-py3-none-any.whl

Keywords: natural-language-processing, text-cleaning, text-preprocessing, text-normalization, user-generated-content

Tags

More Linguistic packages