---
id: sentencepiece
version: "0.2.2"
license: Apache-2.0
license_treatment: permissive
maintenance: active
---

# sentencepiece — Unsupervised text tokenizer and detokenizer.

License: permissive · Maintenance: active · Popularity: top 1,000 on PyPI

## Install

pip install sentencepiece
uv add sentencepiece
poetry add sentencepiece

## Description

# SentencePiece Python Wrapper

Python wrapper for SentencePiece. This API supports the encoding, decoding, and training of SentencePiece models.

For a detailed feature and API comparison with Hugging Face Tokenizers and OpenAI's tiktoken, see the [Tokenizer Comparison Cheat Sheet](tokenizer_comparison_cheat_sheet.md).

## Installation

For Linux (x86_64/aarch64), macOS, and Windows (x64/arm64) environments, you can use the pip command to install the SentencePiece Python module.

```bash
pip install sentencepiece
```

## Basic Usage

The `SentencePieceProcessor` class provides the primary interface for text tokenization (encoding) and detokenization (decoding).

#### Core Methods
*   **`sp.encode(...)`**: Segments input text into token IDs, string pieces, or other formats (like NumPy arrays or Protobuf messages).
*   **`sp.decode(...)`**: Reconstructs the original text from token IDs or string pieces.

#### Input Types & Batch Processing
Both methods support polymorphic inputs and can execute in either single or batch mode:
*   **Single Input**: Pass a single Unicode `str` or raw `bytes` (for encoding), or a single list of IDs/pieces or a 1D NumPy array (for decoding).
*...

## AI interpretation — verify before relying

SentencePiece is a Python wrapper for unsupervised text tokenization and detokenization, supporting encoding text into token IDs or subword pieces and decoding them back, with batch processing and multiple output formats including NumPy arrays and Protobuf messages.

Verdict: SentencePiece is a production-grade, actively maintained tokenizer widely used in NLP pipelines. No known vulnerabilities, permissive Apache-2.0 license, and prebuilt wheels for major platforms minimize friction. Medium install complexity is offset by broad platform coverage and zero runtime dependencies.

[View on SkillFed](https://skillfed.io/packages/sentencepiece) · [View on PyPI](https://pypi.org/project/sentencepiece/)