sacrebleu

What it is and what it does

SacreBLEU wraps the original BLEU reference implementation with added conveniences for machine translation research. It computes BLEU, chrF, chrF++, and TER metrics on detokenized text, handling tokenization automatically according to WMT standards. The package automatically downloads and manages common test sets (like wmt14, wmt17, wmt21) so you can score against them by name rather than hunting down local files. It outputs results with a version signature that documents exactly how the score was computed, making cross-paper comparisons straightforward.

The tool works both as a command-line utility and as a Python library. It supports multiple tokenizers including language-specific ones for Japanese and Chinese, performs statistical significance testing via paired bootstrap resampling and approximate randomization, and defaults to JSON output (with a text format option for backward compatibility). Dependencies include portalocker, regex, tabulate, numpy, colorama, and lxml for various scoring and output formatting tasks.

Use it for:

• Score machine translation system outputs against standard WMT benchmarks without manually downloading or preprocessing test sets
• Compare BLEU scores across different papers and systems using the reproducible version signature to ensure methodological consistency
• Evaluate translation quality using multiple metrics (BLEU, chrF, TER) in a single command with consistent tokenization
• Perform statistical significance testing on paired translation outputs to determine if improvements are meaningful
• Integrate MT evaluation into research pipelines via the Python API for automated scoring of experimental outputs

Worth the install?

AI-flagged interpretation of the facts on this page — verify before relying

SacreBLEU computes BLEU, chrF, and TER scores for machine translation evaluation with automatic test set management and reproducible, comparable results across systems.

Yes. SacreBLEU is actively maintained, has no security vulnerabilities, installs with low friction, and is the standard tool for reproducible machine translation evaluation in the research community. The Apache-2.0 license poses no restrictions. Install it if you work with machine translation evaluation, benchmark scoring, or need comparable BLEU/chrF/TER metrics across systems.

Install

sacrebleu on PyPI

pip install sacrebleu

uv add sacrebleu

poetry add sacrebleu

Installing sacrebleu

Before you install

Low friction installation with six runtime dependencies already packaged as a wheel. Active maintenance with last commit on 2026-07-17 and a release on 2026-01-12; no security vulnerabilities reported.

License in practice

Apache-2.0 permissive license allows use in commercial and proprietary projects without copyleft obligations.

Quickstart

pip install sacrebleu

from sacrebleu import BLEU
bleu = BLEU()
score = bleu.corpus_score(hypotheses, [references])
print(score.score)

Requires Python 3.9 or later. Japanese and Korean tokenizer support requires optional dependencies installed via sacrebleu[ja] or sacrebleu[ko].

Verify before relying

• Whether the package's Python API (corpus_score, etc.) is documented or stable across versions
• Performance characteristics when scoring large translation corpora
• Whether all WMT test sets mentioned in the description are currently available for download

Package facts

Evidence: sacrebleu-2.6.0-py3-none-any.whl

Keywords: machine translation, evaluation, NLP, natural language processing, computational linguistics

Tags

More Scientific/Engineering packages