{"categories":[{"label":"Text Processing","url":"https://skillfed.io/packages/category/text-processing"}],"enrichment":{"capability":"Detects and fixes mojibake (garbled Unicode text caused by encoding mismatches) and recovers correctly-encoded text from multiple layers of encoding corruption.","skillfed_tags":["text-repair","encoding-recovery","nlp-preprocessing"],"use_cases":["Clean scraped web content or user-generated text that has been corrupted by encoding mismatches during storage or transmission.","Preprocess text datasets for NLP research or machine learning to remove mojibake before training.","Repair legacy data imported from systems that mixed character encodings (e.g., UTF-8 decoded as Latin-1).","Decode HTML entities that appear outside HTML context, including non-standard capitalizations.","Fix text with multiple overlapping encoding errors that cannot be solved by a single decode operation."],"what_it_does":"ftfy detects and repairs mojibake\u2014text that was encoded as UTF-8 but decoded as a different encoding (or multiple times in succession)\u2014by recognizing telltale byte patterns and recovering the original string. It handles complex cases including multiple layers of corruption, curly quotes applied over mojibake, non-breaking spaces mangled into regular spaces, and incorrectly capitalized HTML entities. The package is conservative: it avoids false positives by refusing to \"fix\" text that is already sensible, even if it could theoretically be reinterpreted as mojibake.\n\nThe library is used as a data-cleaning step in NLP research and text processing pipelines. It exposes a simple API (primarily `fix_text()` and `fix_encoding()`) and includes command-line tools. It depends only on wcwidth for character width calculations and supports current Python versions (3.9+).","worth_installing":"Yes, if you work with text from diverse or legacy sources. ftfy solves a real, hard problem (mojibake recovery) that few other tools address. Low install friction, no security issues, and active maintenance make it a safe dependency. The Apache license requires attribution but is otherwise permissive. Install it when text corruption is a known issue in your pipeline; skip it if your text is already clean."},"id":"ftfy","links":{"html":"https://skillfed.io/packages/ftfy","md":"https://skillfed.io/packages/ftfy.md","pypi":"https://pypi.org/project/ftfy/"},"maintenance":{"status":"dormant"},"meta":{"latest_release":"2024-10-26","license_spdx":null,"license_treatment":"permissive","name":"ftfy","python_support":"supports_current","summary":"Fixes mojibake and other problems with Unicode, after the fact"},"popularity":{"monthly_downloads":14452490,"position":1228,"tier":"top_5000"},"security":{"n_vulnerabilities":0},"version":"6.3.1"}