sacremoses
SacreMoses
Decision gist · record as of 2026-08-14
Yes, with conditions. Sacremoses is a stable, permissively licensed tool for standard text preprocessing in NLP workflows. Install it if you need Moses-style tokenization or truecasing for machine translation or similar tasks. However, the aging maintenance status means you should verify compatibility with your current Python and dependency versions before relying on it for production work.AI-flagged interpretation of the facts on this page — verify before relying
Before you install
- Requires Python 3.8 or later; sacremoses>=0.0.41 dropped Python 2 support.
- Low friction installation with a pure Python wheel.
- Maintenance is aging—last release was 2023-10-30—but the repository remains active with recent commits and no archived status.
License · maintenance · safety
permissive license (permissive) — MIT License permits commercial and private use with minimal restrictions, making it safe for most projects.
last release 2023-10-30 (1019 days) · last repo commit 2026-02-06 · 497 stars
0 known vulnerabilities (OSV.dev, 2026-08-14) · 2,913,037 downloads/mo, #2,825 on PyPI
Alternatives
Verify before relying
from sacremoses import MosesTokenizer, MosesDetokenizer
mt = MosesTokenizer(lang='en')
tokenized = mt.tokenize('This is a test.', return_str=True)
md = MosesDetokenizer(lang='en')
detokenized = md.detokenize(tokenized.split())- Whether the aging maintenance status affects stability or compatibility with current NLP workflows.
- Performance characteristics when processing large text corpora with joblib parallelization.
What it is and what it does
Sacremoses is a Python implementation of the Moses statistical machine translation toolkit's text processing utilities. It provides language-aware tokenization (splitting text into words and punctuation), detokenization (reconstructing text from tokens), truecasing (restoring proper capitalization from all-caps or all-lowercase text), and punctuation normalization. The package wraps the core Moses algorithms and exposes them through both a Python API and command-line interface.
The library is designed for NLP and machine translation preprocessing pipelines. It supports multiple languages through language-specific rules, offers parallel processing via joblib, and can be used programmatically or as a CLI tool with pipeline chaining. Its four main runtime dependencies—regex, click, joblib, and tqdm—handle pattern matching, command-line interfaces, parallel execution, and progress reporting respectively.
Use it for
- Tokenize raw text for machine translation or NLP model training pipelines.
- Restore proper capitalization in all-caps or ASR output using trained truecasing models.
- Normalize punctuation and control characters in multilingual text datasets.
- Build preprocessing pipelines via CLI with language-specific rules and parallel processing.
- Detokenize model output back into readable text for evaluation or display.
Worth the install?
AI-flagged interpretation of the facts on this page. Verify before relying on it.
Yes, with conditions.
Sacremoses is a stable, permissively licensed tool for standard text preprocessing in NLP workflows. Install it if you need Moses-style tokenization or truecasing for machine translation or similar tasks. However, the aging maintenance status means you should verify compatibility with your current Python and dependency versions before relying on it for production work.
Install
sacremoses on PyPI
Before you install
Low friction installation with a pure Python wheel. Maintenance is aging—last release was 2023-10-30—but the repository remains active with recent commits and no archived status.
Requires Python 3.8 or later; sacremoses>=0.0.41 dropped Python 2 support.
License in practice
MIT License permits commercial and private use with minimal restrictions, making it safe for most projects.
Quickstart
from sacremoses import MosesTokenizer, MosesDetokenizer
mt = MosesTokenizer(lang='en')
tokenized = mt.tokenize('This is a test.', return_str=True)
md = MosesDetokenizer(lang='en')
detokenized = md.detokenize(tokenized.split())Verify before relying
- Whether the aging maintenance status affects stability or compatibility with current NLP workflows.
- Performance characteristics when processing large text corpora with joblib parallelization.
Package facts
| License | permissive license permissive |
| Python support | Supports the current Python release >=3.8 |
| Install friction | Low. Pure-Python wheel |
| Runtime dependencies | 4 packagesregexclickjoblibtqdm |
| Maintenance | Aging 1,019 days since the last release |
| Last repo commit | |
| First released | |
| Downloads | 2,913,037 / month, #2,825 on PyPI 30-day window, as of 2026-08-14 |
| Known vulnerabilities | None known OSV.dev, checked 2026-08-14 |
| Classifiers | License :: OSI Approved :: MIT LicenseOperating System :: OS IndependentProgramming Language :: Python :: 3 |
Evidence: sacremoses-0.1.1-py3-none-any.whl
Tags
Let your AI agent find packages like this
Example. Real query, live index.
You found this page by searching. An agent finds it by wishing: SkillFed indexes 14,416 PyPI packages by what they can do, searchable in plain language.
wish › “text tokenization and detokenization”
- sacremosesSacremoses provides tokenization, detokenization, truecasing, and…
- openvino-tokenizersConverts HuggingFace tokenizers into OpenVINO models for text…
- indic-nlp-libraryIndic NLP Library provides text processing and natural language…
Give your agent the search over MCP, or paste the wish link into any chat.
More Linguistic packages
Detects and normalizes text encoding from unknown or ambiguous sources, supporting all IANA character sets that Python's core library provides codecs for, with the ability to register custom codecs.
tiktoken is a fast BPE tokenizer that converts text into token sequences compatible with OpenAI models, supporting multiple encoding schemes including o200k_base and model-specific encodings.
Install it if you work with OpenAI APIs or need to understand token boundaries in GPT-family models.
Detects character encoding and language in byte sequences with high accuracy, supporting 99 encodings and returning confidence scores, language tags, and MIME types.
Install it if you need to detect character encoding or language in byte data; the rewrite makes it substantially faster and more accurate than its predecessors.
Converts Unicode text to ASCII by transliterating non-ASCII characters into their closest ASCII equivalents, with no runtime dependencies.
However, if transliteration quality or ongoing maintenance matters, consider unidecode instead despite its GPL-only license.
Lark is a parsing library that builds abstract syntax trees from context-free grammars, supporting multiple parsing algorithms (Earley, LALR(1), CYK) with automatic line and column tracking.
Python bindings to the tree-sitter parsing library, enabling incremental parsing and syntax tree analysis for source code.
See also tokenizer · untokenize · normality · jieba3k · razdel · segments · indic-nlp-library · semantic-text-splitter · match · curated-tokenizers