sacrebleu
Hassle-free computation of shareable, comparable, and reproducible BLEU, chrF, and TER scores
Decision gist · record as of 2026-08-14
Yes. SacreBLEU is actively maintained, has no security vulnerabilities, installs with low friction, and is the standard tool for reproducible machine translation evaluation in the research community. The Apache-2.0 license poses no restrictions. Install it if you work with machine translation evaluation, benchmark scoring, or need comparable BLEU/chrF/TER metrics across systems.AI-flagged interpretation of the facts on this page — verify before relying
Before you install
- Requires Python 3.9 or later.
- Japanese and Korean tokenizer support requires optional dependencies installed via sacrebleu[ja] or sacrebleu[ko].
- Low friction installation with six runtime dependencies already packaged as a wheel.
License · maintenance · safety
Apache-2.0 (permissive) — Apache-2.0 permissive license allows use in commercial and proprietary projects without copyleft obligations.
last release 2026-01-12 (214 days) · last repo commit 2026-07-17 · 1,257 stars
0 known vulnerabilities (OSV.dev, 2026-08-14) · 4,139,155 downloads/mo, #2,372 on PyPI
Alternatives
Verify before relying
pip install sacrebleu
from sacrebleu import BLEU
bleu = BLEU()
score = bleu.corpus_score(hypotheses, [references])
print(score.score)- Whether the package's Python API (corpus_score, etc.) is documented or stable across versions
- Performance characteristics when scoring large translation corpora
- Whether all WMT test sets mentioned in the description are currently available for download
What it is and what it does
SacreBLEU wraps the original BLEU reference implementation with added conveniences for machine translation research. It computes BLEU, chrF, chrF++, and TER metrics on detokenized text, handling tokenization automatically according to WMT standards. The package automatically downloads and manages common test sets (like wmt14, wmt17, wmt21) so you can score against them by name rather than hunting down local files. It outputs results with a version signature that documents exactly how the score was computed, making cross-paper comparisons straightforward.
The tool works both as a command-line utility and as a Python library. It supports multiple tokenizers including language-specific ones for Japanese and Chinese, performs statistical significance testing via paired bootstrap resampling and approximate randomization, and defaults to JSON output (with a text format option for backward compatibility). Dependencies include portalocker, regex, tabulate, numpy, colorama, and lxml for various scoring and output formatting tasks.
Use it for
- Score machine translation system outputs against standard WMT benchmarks without manually downloading or preprocessing test sets
- Compare BLEU scores across different papers and systems using the reproducible version signature to ensure methodological consistency
- Evaluate translation quality using multiple metrics (BLEU, chrF, TER) in a single command with consistent tokenization
- Perform statistical significance testing on paired translation outputs to determine if improvements are meaningful
- Integrate MT evaluation into research pipelines via the Python API for automated scoring of experimental outputs
Worth the install?
AI-flagged interpretation of the facts on this page. Verify before relying on it.
Yes.
SacreBLEU is actively maintained, has no security vulnerabilities, installs with low friction, and is the standard tool for reproducible machine translation evaluation in the research community. The Apache-2.0 license poses no restrictions. Install it if you work with machine translation evaluation, benchmark scoring, or need comparable BLEU/chrF/TER metrics across systems.
Install
sacrebleu on PyPI
Before you install
Low friction installation with six runtime dependencies already packaged as a wheel. Active maintenance with last commit on 2026-07-17 and a release on 2026-01-12; no security vulnerabilities reported.
Requires Python 3.9 or later. Japanese and Korean tokenizer support requires optional dependencies installed via sacrebleu[ja] or sacrebleu[ko].
License in practice
Apache-2.0 permissive license allows use in commercial and proprietary projects without copyleft obligations.
Quickstart
pip install sacrebleu
from sacrebleu import BLEU
bleu = BLEU()
score = bleu.corpus_score(hypotheses, [references])
print(score.score)Verify before relying
- Whether the package's Python API (corpus_score, etc.) is documented or stable across versions
- Performance characteristics when scoring large translation corpora
- Whether all WMT test sets mentioned in the description are currently available for download
Package facts
| License | Apache-2.0 permissive |
| Python support | Supports the current Python release >=3.9 |
| Install friction | Low. Pure-Python wheel |
| Runtime dependencies | 6 packagesportalockerregextabulatenumpycoloramalxml |
| Maintenance | Actively maintained 214 days since the last release |
| Last repo commit | |
| First released | |
| Downloads | 4,139,155 / month, #2,372 on PyPI 30-day window, as of 2026-08-14 |
| Known vulnerabilities | None known OSV.dev, checked 2026-08-14 |
| Classifiers | Development Status :: 5 - Production/StableIntended Audience :: DevelopersIntended Audience :: Science/ResearchOperating System :: MacOS :: MacOS XOperating System :: Microsoft :: WindowsOperating System :: POSIXProgramming Language :: Python :: 3 :: OnlyTopic :: Scientific/EngineeringTopic :: Scientific/Engineering :: Artificial IntelligenceTopic :: Text ProcessingTyping :: Typed |
Evidence: sacrebleu-2.6.0-py3-none-any.whl
Tags
Let your AI agent find packages like this
Example. Real query, live index.
You found this page by searching. An agent finds it by wishing: SkillFed indexes 14,416 PyPI packages by what they can do, searchable in plain language.
wish › “BLEU score computation”
- sacrebleuSacreBLEU computes BLEU, chrF, and TER scores for machine translation…
- pycocoevalcapEvaluates image captions using multiple automatic metrics (BLEU,…
- rouge-scoreComputes ROUGE scores (ROUGE-N, ROUGE-L, ROUGE-Lsum) to evaluate the…
Give your agent the search over MCP, or paste the wish link into any chat.
More Scientific/Engineering packages
NumPy provides an N-dimensional array object and a comprehensive suite of mathematical, linear algebra, Fourier transform, and random number functions for scientific computing in Python.
pandas provides fast, flexible data structures (Series and DataFrame) for loading, cleaning, transforming, and analyzing labeled or relational data in Python.
scipy provides numerical algorithms for mathematics, science, and engineering—including optimization, integration, linear algebra, Fourier transforms, signal and image processing, and ODE solvers—built on numpy arrays.
scikit-learn provides a comprehensive Python library for supervised and unsupervised machine learning, including classification, regression, clustering, dimensionality reduction, and model evaluation tools built on NumPy and SciPy.
Install it if you need to train, evaluate, or deploy supervised or unsupervised learning models.
dill extends Python's pickle module to serialize and deserialize a much wider range of Python objects, including functions, lambdas, classes, and interpreter sessions, to byte streams for storage or network transmission.
Multiprocess is an enhanced fork of Python's standard multiprocessing library that uses dill for better serialization, allowing you to spawn processes with a threading-like API and share complex objects between them.
Install it if you use multiprocessing and encounter pickle serialization limits with lambdas or complex objects.
See also unbabel-comet · rouge-chinese · image-similarity-measures · rouge · pycocoevalcap · mir-eval · strands-agents-evals · dtlpymetrics · rouge-score · rouge-metric