langid

What it is and what it does

langid.py is a standalone language identification tool that classifies text into one of 97 pre-trained languages using a statistical model. It operates in three modes: as a command-line tool for interactive or batch processing, as a Python library for programmatic use, and as a WSGI-compliant web service. The model was trained on diverse sources including Wikipedia, Reuters, and ClueWeb data, and is designed to be fast and robust to domain-specific features like HTML markup.

The package requires only Python 2.7+ (or Python 3) and numpy. It returns both a language code (ISO 639-1) and a confidence score in log-probability space, with optional normalization to the 0–1 range. You can constrain predictions to a specific set of languages and process input from stdin, files, or HTTP requests. However, the project has been abandoned since 2016 with no active maintenance, so it may not reflect modern language usage or receive bug fixes.

Use it for:

• Classify user-submitted text by language in a web application or API without external service calls.
• Batch process document collections to sort or filter by detected language across 97 supported languages.
• Deploy as a lightweight WSGI web service for language detection in microservice architectures.
• Constrain language detection to a known subset (e.g., en, de, fr) to improve accuracy in multilingual but bounded scenarios.
• Identify language in log files or text streams line-by-line for monitoring or data pipeline preprocessing.

Worth the install?

AI-flagged interpretation of the facts on this page — verify before relying

Identifies the language of text input across 97 languages using a pre-trained statistical model, available as a command-line tool, Python library, or WSGI web service.

Yes, if you need offline language detection across many languages and can accept an unmaintained codebase. The package is stable, carries no known vulnerabilities, and has a permissive license. However, do not use it if you require active maintenance, support for modern Python versions beyond basic compatibility, or confidence that the model reflects current language patterns. For new projects, consider actively maintained alternatives.

Install

langid on PyPI

pip install langid

uv add langid

poetry add langid

Installing langid

Before you install

High install friction due to numpy dependency and no active maintenance since 2016. The package is marked abandoned with no commits since 2020-01-01, though it remains in production/stable status.

License in practice

BSD license is permissive, allowing commercial and private use with minimal restrictions—suitable for most projects that can accept a mature, unmaintained codebase.

Quickstart

pip install langid
import langid
result = langid.classify("This is a test")
print(result)  # ('en', -54.41310358047485)

Requires numpy; Python 2.7+ or Python 3 support is present but training tools are Python 2-only.

Verify before relying

• Whether the pre-trained model on 97 languages remains accurate for modern text and evolving language usage patterns.
• Performance characteristics (speed, memory) on large-scale batch processing or real-time web service deployments.
• Compatibility with current Python 3 versions and whether the package works reliably on modern systems.

Package facts

Evidence: langid-1.1.6.tar.gz

Keywords: language, detection

Tags

More Artificial Intelligence packages