{"categories":[{"label":"Linguistic","url":"https://skillfed.io/packages/category/text-processing-linguistic/2"}],"enrichment":{"capability":"BPEmb provides pre-trained subword embeddings in 275 languages based on Byte-Pair Encoding, enabling subword segmentation and embedding lookup for neural NLP models.","skillfed_tags":["multilingual-nlp","embeddings","tokenization"],"use_cases":["Tokenize text into subwords for morphologically rich or low-resource languages where word-level tokenization is insufficient.","Initialize neural sequence models with pre-trained multilingual embeddings without training your own.","Build cross-lingual NLP systems that share a common subword representation across languages.","Experiment with different vocabulary sizes to balance granularity and sparsity for a specific language.","Embed text fragments for similarity search or clustering in languages beyond English."],"what_it_does":"BPEmb is a collection of pre-trained subword embeddings covering 275 languages, built on Wikipedia using Byte-Pair Encoding (BPE). It wraps gensim's KeyedVectors to provide both subword tokenization and embedding vectors for use in neural NLP pipelines.\n\nThe package serves two main purposes: splitting text into subwords according to a learned vocabulary (with configurable vocabulary sizes from 1000 to 200000 tokens), and providing dense vector representations for those subwords. You load a language-specific model, encode text into subword tokens or token IDs, and either retrieve embeddings directly or use the built-in embed method. Models and embeddings download automatically on first use.","worth_installing":"Yes, if you need multilingual subword embeddings and can accept a dormant package. The low install friction, permissive license, and broad language coverage make it useful for research and prototyping. However, the 682-day gap since the last release means no bug fixes or updates are forthcoming; verify that the remote model repository remains accessible before relying on it in production."},"id":"bpemb","links":{"html":"https://skillfed.io/packages/bpemb","md":"https://skillfed.io/packages/bpemb.md","pypi":"https://pypi.org/project/bpemb/"},"maintenance":{"status":"dormant"},"meta":{"latest_release":"2024-10-01","license_spdx":null,"license_treatment":"permissive","name":"bpemb","python_support":"unspecified","summary":"Byte-pair embeddings in 275 languages"},"popularity":{"monthly_downloads":124620,"position":11860,"tier":"top_15000"},"security":{"n_vulnerabilities":0},"version":"0.3.6"}