bpemb

What it is and what it does

BPEmb is a collection of pre-trained subword embeddings covering 275 languages, built on Wikipedia using Byte-Pair Encoding (BPE). It wraps gensim's KeyedVectors to provide both subword tokenization and embedding vectors for use in neural NLP pipelines.

The package serves two main purposes: splitting text into subwords according to a learned vocabulary (with configurable vocabulary sizes from 1000 to 200000 tokens), and providing dense vector representations for those subwords. You load a language-specific model, encode text into subword tokens or token IDs, and either retrieve embeddings directly or use the built-in embed method. Models and embeddings download automatically on first use.

Use it for:

• Tokenize text into subwords for morphologically rich or low-resource languages where word-level tokenization is insufficient.
• Initialize neural sequence models with pre-trained multilingual embeddings without training your own.
• Build cross-lingual NLP systems that share a common subword representation across languages.
• Experiment with different vocabulary sizes to balance granularity and sparsity for a specific language.
• Embed text fragments for similarity search or clustering in languages beyond English.

Worth the install?

AI-flagged interpretation of the facts on this page — verify before relying

BPEmb provides pre-trained subword embeddings in 275 languages based on Byte-Pair Encoding, enabling subword segmentation and embedding lookup for neural NLP models.

Yes, if you need multilingual subword embeddings and can accept a dormant package. The low install friction, permissive license, and broad language coverage make it useful for research and prototyping. However, the 682-day gap since the last release means no bug fixes or updates are forthcoming; verify that the remote model repository remains accessible before relying on it in production.

Install

bpemb on PyPI

pip install bpemb

uv add bpemb

poetry add bpemb

Installing bpemb

Before you install

Low install friction with a pure Python wheel and five common dependencies (gensim, numpy, requests, sentencepiece, tqdm). The package is dormant—last release 682 days ago—so expect no active maintenance or updates.

License in practice

MIT license is permissive; you can use BPEmb freely in commercial and proprietary projects with minimal restrictions.

Quickstart

pip install bpemb

from bpemb import BPEmb
bpemb_en = BPEmb(lang="en", dim=50)
print(bpemb_en.encode("Stratford"))
# Models and embeddings download automatically on first use

Models and embeddings download automatically from nlp.h-its.org on first use; requires internet access and disk space for language-specific files.

Verify before relying

• Whether the remote model repository (nlp.h-its.org) remains stable and available long-term given the package's dormant status.
• Performance characteristics when working with very large texts or batch processing across many languages.

Package facts

Evidence: bpemb-0.3.6-py3-none-any.whl

Tags

More Linguistic packages