ftfy

What it is and what it does

ftfy detects and repairs mojibake—text that was encoded as UTF-8 but decoded as a different encoding (or multiple times in succession)—by recognizing telltale byte patterns and recovering the original string. It handles complex cases including multiple layers of corruption, curly quotes applied over mojibake, non-breaking spaces mangled into regular spaces, and incorrectly capitalized HTML entities. The package is conservative: it avoids false positives by refusing to "fix" text that is already sensible, even if it could theoretically be reinterpreted as mojibake.

The library is used as a data-cleaning step in NLP research and text processing pipelines. It exposes a simple API (primarily `fix_text()` and `fix_encoding()`) and includes command-line tools. It depends only on wcwidth for character width calculations and supports current Python versions (3.9+).

Use it for:

• Clean scraped web content or user-generated text that has been corrupted by encoding mismatches during storage or transmission.
• Preprocess text datasets for NLP research or machine learning to remove mojibake before training.
• Repair legacy data imported from systems that mixed character encodings (e.g., UTF-8 decoded as Latin-1).
• Decode HTML entities that appear outside HTML context, including non-standard capitalizations.
• Fix text with multiple overlapping encoding errors that cannot be solved by a single decode operation.

Worth the install?

AI-flagged interpretation of the facts on this page — verify before relying

Detects and fixes mojibake (garbled Unicode text caused by encoding mismatches) and recovers correctly-encoded text from multiple layers of encoding corruption.

Yes, if you work with text from diverse or legacy sources. ftfy solves a real, hard problem (mojibake recovery) that few other tools address. Low install friction, no security issues, and active maintenance make it a safe dependency. The Apache license requires attribution but is otherwise permissive. Install it when text corruption is a known issue in your pipeline; skip it if your text is already clean.

Install

ftfy on PyPI

pip install ftfy

uv add ftfy

poetry add ftfy

Installing ftfy

Before you install

Low friction: pure Python wheel with a single runtime dependency (wcwidth). Last release was 657 days ago; repo remains active with recent commits and no archived status, though maintenance is dormant.

License in practice

Apache-2.0 permissive license requires attribution to Robyn Speer. The package explicitly prohibits use in AI training datasets or derived works that obscure authorship; violators may be notified and required to remedy or delete copies.

Quickstart

pip install ftfy

from ftfy import fix_text
print(fix_text('âœ" No problems'))
# Output: ✔ No problems

Requires Python 3.9 or later.

Verify before relying

• Whether the package handles all real-world encoding scenarios or only a documented subset of common mojibake patterns.
• Performance characteristics on very large text volumes or streaming input.

Package facts

Evidence: ftfy-6.3.1-py3-none-any.whl

Tags

More Text Processing packages