bert-score

What it is and what it does

BERTScore is an automatic evaluation metric for text generation tasks that uses contextual embeddings from pre-trained BERT models to compare candidate and reference sentences. Rather than relying on surface-level token overlap, it matches words by cosine similarity in embedding space, producing precision, recall, and F1 scores that correlate better with human judgment than traditional metrics.

The package wraps torch, transformers, and related dependencies to provide both a simple `score()` function and a `BERTScorer` object that caches models across multiple evaluations. It supports about 130 pre-trained models including BERT variants, RoBERTa, DeBERTa, T5, BART, and multilingual options. The default model is roberta-large, though the documentation recommends microsoft/deberta-xlarge-mnli for better correlation with human scores.

Use it for:

• Evaluate machine translation output by scoring candidate translations against reference translations.
• Assess abstractive summarization quality by comparing generated summaries to reference summaries.
• Score image captioning or visual question-answering outputs against reference captions or answers.
• Benchmark text generation models during development without manual human evaluation.
• Compare multiple candidate outputs for the same input and select the highest-scoring one.

Worth the install?

AI-flagged interpretation of the facts on this page — verify before relying

BERTScore computes automatic evaluation metrics for text generation by comparing candidate and reference sentences using contextual embeddings from BERT, producing precision, recall, and F1 scores.

Yes, with conditions. BERTScore is well-established with 1913 GitHub stars and solves a real problem in NLP evaluation. However, the package is dormant (last release 2023-02-20) and computationally heavy—it requires PyTorch and GPU access for practical use, and first-run model downloads are slow. Install it if you need embedding-based text evaluation and can afford the compute cost; avoid it if you need active maintenance or lightweight CPU-only evaluation.

Install

bert-score on PyPI

pip install bert-score

uv add bert-score

poetry add bert-score

Installing bert-score

Before you install

Low install friction with a pure Python wheel. The package is dormant (last release 2023-02-20, 1271 days ago) but has 1913 GitHub stars and remains compatible with current transformers versions as of 0.3.13. Eight runtime dependencies including torch and transformers are substantial but standard for NLP work.

License in practice

MIT license is permissive, allowing commercial and private use with minimal restrictions—suitable for most research and production contexts.

Quickstart

pip install bert-score

from bert_score import score

cands = ["the cat is on the mat"]
refs = [["a cat is on the mat"]]
P, R, F1 = score(cands, refs, lang="en")

Requires PyTorch >= 1.0.0 and Python >= 3.6. First run downloads a pre-trained BERT model (e.g., roberta-large by default), which is computationally intensive and requires GPU for practical use.

Verify before relying

• Whether the package's performance correlations with human evaluation remain current given the 2023 release date and evolving model landscape.
• Practical memory and compute requirements for scoring at scale with different model choices.
• Current monthly download volume and active user base given dormant maintenance status.

Package facts

Evidence: bert_score-0.3.13-py3-none-any.whl

Keywords: BERT, NLP, deep, learning, google, metric

Tags

More Artificial Intelligence packages

Further reading