---
id: bpemb
version: "0.3.6"
license: MIT
license_treatment: permissive
maintenance: dormant
---

# bpemb — Byte-pair embeddings in 275 languages

License: permissive · Maintenance: dormant · Downloads: 124.6K/mo

## What it is and what it does

BPEmb is a collection of pre-trained subword embeddings covering 275 languages, built on Wikipedia using Byte-Pair Encoding (BPE). It wraps gensim's KeyedVectors to provide both subword tokenization and embedding vectors for use in neural NLP pipelines.

The package serves two main purposes: splitting text into subwords according to a learned vocabulary (with configurable vocabulary sizes from 1000 to 200000 tokens), and providing dense vector representations for those subwords. You load a language-specific model, encode text into subword tokens or token IDs, and either retrieve embeddings directly or use the built-in embed method. Models and embeddings download automatically on first use.

Use it for:

- Tokenize text into subwords for morphologically rich or low-resource languages where word-level tokenization is insufficient.
- Initialize neural sequence models with pre-trained multilingual embeddings without training your own.
- Build cross-lingual NLP systems that share a common subword representation across languages.
- Experiment with different vocabulary sizes to balance granularity and sparsity for a specific language.
- Embed text fragments for similarity search or clustering in languages beyond English.

## Worth the install?

AI-flagged interpretation of the facts above — verify before relying.

BPEmb provides pre-trained subword embeddings in 275 languages based on Byte-Pair Encoding, enabling subword segmentation and embedding lookup for neural NLP models.

Yes, if you need multilingual subword embeddings and can accept a dormant package. The low install friction, permissive license, and broad language coverage make it useful for research and prototyping. However, the 682-day gap since the last release means no bug fixes or updates are forthcoming; verify that the remote model repository remains accessible before relying on it in production.

## Install

pip install bpemb
uv add bpemb
poetry add bpemb

## Installing bpemb

Before you install: Low install friction with a pure Python wheel and five common dependencies (gensim, numpy, requests, sentencepiece, tqdm). The package is dormant—last release 682 days ago—so expect no active maintenance or updates.

License in practice: MIT license is permissive; you can use BPEmb freely in commercial and proprietary projects with minimal restrictions.

Quickstart:

pip install bpemb

from bpemb import BPEmb
bpemb_en = BPEmb(lang="en", dim=50)
print(bpemb_en.encode("Stratford"))
# Models and embeddings download automatically on first use

Models and embeddings download automatically from nlp.h-its.org on first use; requires internet access and disk space for language-specific files.

Verify before relying:

- Whether the remote model repository (nlp.h-its.org) remains stable and available long-term given the package's dormant status.
- Performance characteristics when working with very large texts or batch processing across many languages.

## Package facts

- License: MIT (permissive)
- Python support: unspecified
- Install friction: low
- Maintenance: dormant
- Downloads: 124.6K/month (top 15,000 on PyPI)
- Known vulnerabilities: none known

## Tags

multilingual subword embeddings, byte-pair encoding BPE, pre-trained language embeddings, subword tokenization, cross-lingual NLP embeddings, Wikipedia-trained embeddings, neural language model input, multilingual-nlp, embeddings, tokenization

[View on SkillFed](https://skillfed.io/packages/bpemb) · [View on PyPI](https://pypi.org/project/bpemb/)