sentencepiece

Unsupervised text tokenizer and detokenizer.

sentencepiece Permissive license Apache-2.0 Active ★ 12,021 v0.2.2 released 2026-07-12

Install

sentencepiece on PyPI

pip

pip install sentencepiece

uv

uv add sentencepiece

poetry

poetry add sentencepiece

Package facts

License	Apache-2.0 (permissive)
Python support	supports the current Python release (>=3.9)
Install friction	medium — platform-specific wheel
Runtime dependencies	none
Maintenance	actively maintained — 32 days since the last release
Last repo commit	2026-08-13
First released	2017-08-28
Popularity	one of the top 1,000 most-downloaded packages on PyPI (30-day window, as of 2026-08-13)
Known vulnerabilities	none known (OSV.dev, checked 2026-08-13)

Evidence: sentencepiece-0.2.2-cp310-cp310-macosx_10_9_universal2.whl; sentencepiece-0.2.2-cp310-cp310-macosx_10_9_x86_64.whl; sentencepiece-0.2.2-cp310-cp310-macosx_11_0_arm64.whl; sentencepiece-0.2.2-cp310-cp310-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl; sentencepiece-0.2.2-cp310-cp310-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl; sentencepiece-0.2.2-cp310-cp310-win_amd64.whl; sentencepiece-0.2.2-cp310-cp310-win_arm64.whl; sentencepiece-0.2.2-cp311-cp311-macosx_10_9_universal2.whl; sentencepiece-0.2.2-cp311-cp311-macosx_10_9_x86_64.whl; sentencepiece-0.2.2-cp311-cp311-macosx_11_0_arm64.whl; sentencepiece-0.2.2-cp311-cp311-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl; sentencepiece-0.2.2-cp311-cp311-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl; sentencepiece-0.2.2-cp311-cp311-win_amd64.whl; sentencepiece-0.2.2-cp311-cp311-win_arm64.whl; sentencepiece-0.2.2-cp312-cp312-macosx_10_13_universal2.whl; sentencepiece-0.2.2-cp312-cp312-macosx_10_13_x86_64.whl; sentencepiece-0.2.2-cp312-cp312-macosx_11_0_arm64.whl; sentencepiece-0.2.2-cp312-cp312-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl; sentencepiece-0.2.2-cp312-cp312-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl; sentencepiece-0.2.2-cp312-cp312-win_amd64.whl

Development Status :: 5 - Production/StableEnvironment :: ConsoleIntended Audience :: DevelopersIntended Audience :: Science/ResearchOperating System :: MacOS :: MacOS XOperating System :: Microsoft :: WindowsOperating System :: POSIX :: LinuxProgramming Language :: PythonProgramming Language :: Python :: 3Programming Language :: Python :: 3.10Programming Language :: Python :: 3.11Programming Language :: Python :: 3.12Programming Language :: Python :: 3.13Programming Language :: Python :: 3.14Programming Language :: Python :: 3.9Programming Language :: Python :: Free Threading :: 2 - BetaTopic :: Software Development :: Libraries :: Python ModulesTopic :: Text Processing :: Linguistic

About sentencepiece

from the package's own PyPI description — quoted content, verbatim

SentencePiece Python Wrapper

Python wrapper for SentencePiece. This API supports the encoding, decoding, and training of SentencePiece models.

For a detailed feature and API comparison with Hugging Face Tokenizers and OpenAI's tiktoken, see the Tokenizer Comparison Cheat Sheet.

Installation

For Linux (x86_64/aarch64), macOS, and Windows (x64/arm64) environments, you can use the pip command to install the SentencePiece Python module.

pip install sentencepiece

Basic Usage

The SentencePieceProcessor class provides the primary interface for text tokenization (encoding) and detokenization (decoding).

Core Methods

sp.encode(...): Segments input text into token IDs, string pieces, or other formats (like NumPy arrays or Protobuf messages).
sp.decode(...): Reconstructs the original text from token IDs or string pieces.

Input Types & Batch Processing

Both methods support polymorphic inputs and can execute in either single or batch mode: * Single Input: Pass a single Unicode str or raw bytes (for encoding), or a single list of IDs/pieces or a 1D NumPy array (for decoding). *...

Read as markdown · JSON record · Source repository · Homepage

AI interpretation — verify before relying

AI-generated interpretation of the package facts above; every digit, version, license, or vulnerability id it cites is grounded in the facts already shown on this page

SentencePiece is a Python wrapper for unsupervised text tokenization and detokenization, supporting encoding text into token IDs or subword pieces and decoding them back, with batch processing and multiple output formats including NumPy arrays and Protobuf messages.

Medium install friction due to compiled wheels; prebuilt binaries available for Python 3.9–3.12 on Linux (x86_64/aarch64), macOS, and Windows (x64/arm64). Actively maintained with recent release (32 days old) and strong community signal (12021 GitHub stars).

Licensed under Apache-2.0 (permissive), allowing commercial and private use with minimal restrictions; attribution required but no copyleft obligations.

Usage

import sentencepiece as spm

sp = spm.SentencePieceProcessor(model_file='model.model')
ids = sp.encode('This is a test')
text = sp.decode(ids)
print(text)

Requires a trained SentencePiece model file (.model) to be available; the package itself does not include pre-trained models.

Verdict: SentencePiece is a production-grade, actively maintained tokenizer widely used in NLP pipelines. No known vulnerabilities, permissive Apache-2.0 license, and prebuilt wheels for major platforms minimize friction. Medium install complexity is offset by broad platform coverage and zero runtime dependencies.

Needs verification

Whether pre-trained model files are distributed separately or must be obtained from external sources
Performance characteristics and memory overhead for large-scale batch encoding operations
Compatibility with free-threaded Python 3.13+ (classifier indicates beta support)

text tokenization subwordsentence piece tokenizerencode decode text tokensnlp tokenization librarybatch text encodingvocabulary segmentationlanguage model preprocessing