py3langid

What it is and what it does

py3langid is a modernized fork of langid.py that detects which of 97 languages a piece of text is written in. It returns both the ISO 639-1 language code and a confidence score. The package is designed to be a drop-in replacement for the original, with significant performance improvements: imports are about 30% faster, model loading is 25-30x faster, and classification itself is 5-6x faster on paragraphs.

The tool works as both a Python library and a command-line utility. It uses a pre-trained statistical model built from diverse sources (JRC-Acquis, ClueWeb 09, Wikipedia, Reuters RCV2, Debian i18n) and requires only numpy as a dependency. It supports language subsetting, probability normalization, and can be deployed as a WSGI web service.

Use it for:

• Automatically categorize user-generated content by language for multilingual applications or content management systems.
• Filter or route text documents to language-specific processing pipelines in NLP workflows.
• Identify the dominant language in mixed-language documents or social media posts for downstream analysis.
• Build language detection into a command-line data pipeline using the standalone langid tool.
• Normalize confidence scores to probabilities for machine learning feature engineering or confidence thresholding.

Worth the install?

AI-flagged interpretation of the facts on this page — verify before relying

Identifies the language of text in one of 97 languages using a pre-trained statistical model, with optimized speed for Python 3.6+.

Yes, if you need fast language detection across 97 languages with minimal dependencies. The dormant maintenance status is not a blocker—the package is stable, has no known vulnerabilities, and the last commit is recent enough. Install if you want a lightweight, performant drop-in replacement for the original langid.py; skip if you need active development or support for languages outside the pre-trained set.

Install

py3langid on PyPI

pip install py3langid

uv add py3langid

poetry add py3langid

Installing py3langid

Before you install

Low friction: pure Python wheel with only numpy as a runtime dependency. Dormant maintenance (last release 787 days ago, last commit 2024-11-22), but stable and archived repository shows no active development pressure.

License in practice

BSD-3-Clause permissive license allows commercial and private use with minimal restrictions; attribution and license notice required in distributions.

Quickstart

pip install py3langid

import py3langid as langid
lang, prob = langid.classify('This text is in English.')
print(lang, prob)  # ('en', -56.77429)

Requires Python >= 3.8 and numpy; model loading on first use may take a moment.

Verify before relying

• Whether the 30% import speedup and 25-30x model loading speedup claims are reproducible in current environments.
• Accuracy consistency across the 97 supported languages and whether results degrade on mixed-language or code-heavy text.

Package facts

Evidence: py3langid-0.3.0-py3-none-any.whl

Keywords: language detection, language identification, langid, langid.py

Tags

More Artificial Intelligence packages