wordfreq

What it is and what it does

wordfreq is a lookup library for word frequencies across over 40 languages, drawing from multiple corpus sources including Wikipedia, news, books, web text, Twitter, and Reddit. It provides two main query functions: word_frequency() returns a decimal frequency between 0 and 1, while zipf_frequency() returns a logarithmic scale where a Zipf value of 6 means the word appears once per thousand words. The library offers both 'small' wordlists (words appearing at least once per million words) and 'large' lists (once per 100 million words), with a 'best' default that uses large when available.

The package combines frequencies from multiple sources using a figure-skating-inspired method that discards outliers and averages the remainder, then rescales to preserve relative precision. It handles special cases like numbers (aggregating by digit shape and applying Benford's law for first-digit distribution) and supports Unicode text including non-Latin digit systems. Dependencies are lightweight: msgpack, langcodes, regex, ftfy, and locate. It runs on Python 3.8+ and has no known security vulnerabilities.

Use it for:

• Build spell-checkers or autocomplete systems that rank suggestions by word prevalence in a target language.
• Filter or weight text analysis results by word frequency to focus on meaningful rather than common words.
• Estimate language proficiency or corpus difficulty by analyzing the frequency distribution of words in a text.
• Train NLP models with frequency-weighted word embeddings or vocabulary pruning across multiple languages.
• Validate or debug tokenization and language detection by checking whether token frequencies match expected patterns.

Worth the install?

AI-flagged interpretation of the facts on this page — verify before relying

wordfreq looks up word frequencies across over 40 languages from multiple corpus sources, returning frequencies as decimal values or on a logarithmic Zipf scale.

Yes, if you need multilingual word frequency data. The package is stable, well-maintained at a dormant pace (last commit January 2025), has no security issues, and covers a broad range of languages with multiple corpus sources. Install friction is low. The main consideration is whether you need active development—the package is not being enhanced, only kept functional. For lookup-only use cases, this is a solid choice.

Install

wordfreq on PyPI

pip install wordfreq

uv add wordfreq

poetry add wordfreq

Installing wordfreq

Before you install

Low install friction with a pure-Python wheel distribution. Maintenance is dormant—last release was 2023-11-21 and the repository received its most recent commit on 2025-01-04, so the package is not actively developed but remains functional and receives occasional updates.

License in practice

Licensed under Apache-2.0 (permissive), allowing commercial and private use with minimal restrictions beyond attribution and liability disclaimers.

Quickstart

pip install wordfreq

from wordfreq import word_frequency, zipf_frequency

freq = word_frequency('cafe', 'en')
zipf = zipf_frequency('the', 'en')

Requires Python 3.8 or later. Chinese, Japanese, and Korean word frequencies require additional installation steps beyond the base package.

Verify before relying

• Whether the 'large' wordlists are automatically downloaded on first use or require manual setup.
• Memory footprint and load time for different language/wordlist combinations in production.
• How frequently the underlying corpus data is refreshed or updated.

Package facts

Evidence: wordfreq-3.1.1-py3-none-any.whl

Tags

More Linguistic packages