sacremoses

What it is and what it does

Sacremoses is a Python implementation of the Moses statistical machine translation toolkit's text processing utilities. It provides language-aware tokenization (splitting text into words and punctuation), detokenization (reconstructing text from tokens), truecasing (restoring proper capitalization from all-caps or all-lowercase text), and punctuation normalization. The package wraps the core Moses algorithms and exposes them through both a Python API and command-line interface.

The library is designed for NLP and machine translation preprocessing pipelines. It supports multiple languages through language-specific rules, offers parallel processing via joblib, and can be used programmatically or as a CLI tool with pipeline chaining. Its four main runtime dependencies—regex, click, joblib, and tqdm—handle pattern matching, command-line interfaces, parallel execution, and progress reporting respectively.

Use it for:

• Tokenize raw text for machine translation or NLP model training pipelines.
• Restore proper capitalization in all-caps or ASR output using trained truecasing models.
• Normalize punctuation and control characters in multilingual text datasets.
• Build preprocessing pipelines via CLI with language-specific rules and parallel processing.
• Detokenize model output back into readable text for evaluation or display.

Worth the install?

AI-flagged interpretation of the facts on this page — verify before relying

Sacremoses provides tokenization, detokenization, truecasing, and punctuation normalization for text processing, primarily for machine translation and NLP pipelines.

Yes, with conditions. Sacremoses is a stable, permissively licensed tool for standard text preprocessing in NLP workflows. Install it if you need Moses-style tokenization or truecasing for machine translation or similar tasks. However, the aging maintenance status means you should verify compatibility with your current Python and dependency versions before relying on it for production work.

Install

sacremoses on PyPI

pip install sacremoses

uv add sacremoses

poetry add sacremoses

Installing sacremoses

Before you install

Low friction installation with a pure Python wheel. Maintenance is aging—last release was 2023-10-30—but the repository remains active with recent commits and no archived status.

License in practice

MIT License permits commercial and private use with minimal restrictions, making it safe for most projects.

Quickstart

from sacremoses import MosesTokenizer, MosesDetokenizer
mt = MosesTokenizer(lang='en')
tokenized = mt.tokenize('This is a test.', return_str=True)
md = MosesDetokenizer(lang='en')
detokenized = md.detokenize(tokenized.split())

Requires Python 3.8 or later; sacremoses>=0.0.41 dropped Python 2 support.

Verify before relying

• Whether the aging maintenance status affects stability or compatibility with current NLP workflows.
• Performance characteristics when processing large text corpora with joblib parallelization.

Package facts

Evidence: sacremoses-0.1.1-py3-none-any.whl

Tags

More Linguistic packages