bert-score
PyTorch implementation of BERT score
Decision gist · record as of 2026-08-14
Yes, with conditions. BERTScore is well-established with 1913 GitHub stars and solves a real problem in NLP evaluation. However, the package is dormant (last release 2023-02-20) and computationally heavy—it requires PyTorch and GPU access for practical use, and first-run model downloads are slow. Install it if you need embedding-based text evaluation and can afford the compute cost; avoid it if you need active maintenance or lightweight CPU-only evaluation.AI-flagged interpretation of the facts on this page — verify before relying
Before you install
- Requires PyTorch >= 1.0.0 and Python >= 3.6.
- First run downloads a pre-trained BERT model (e.g., roberta-large by default), which is computationally intensive and requires GPU for practical use.
- Low install friction with a pure Python wheel.
License · maintenance · safety
MIT (permissive) — MIT license is permissive, allowing commercial and private use with minimal restrictions—suitable for most research and production contexts.
last release 2023-02-20 (1271 days) · last repo commit 2024-07-30 · 1,913 stars
0 known vulnerabilities (OSV.dev, 2026-08-14) · 673,466 downloads/mo, #5,396 on PyPI
Alternatives
Verify before relying
pip install bert-score
from bert_score import score
cands = ["the cat is on the mat"]
refs = [["a cat is on the mat"]]
P, R, F1 = score(cands, refs, lang="en")- Whether the package's performance correlations with human evaluation remain current given the 2023 release date and evolving model landscape.
- Practical memory and compute requirements for scoring at scale with different model choices.
- Current monthly download volume and active user base given dormant maintenance status.
What it is and what it does
BERTScore is an automatic evaluation metric for text generation tasks that uses contextual embeddings from pre-trained BERT models to compare candidate and reference sentences. Rather than relying on surface-level token overlap, it matches words by cosine similarity in embedding space, producing precision, recall, and F1 scores that correlate better with human judgment than traditional metrics.
The package wraps torch, transformers, and related dependencies to provide both a simple `score()` function and a `BERTScorer` object that caches models across multiple evaluations. It supports about 130 pre-trained models including BERT variants, RoBERTa, DeBERTa, T5, BART, and multilingual options. The default model is roberta-large, though the documentation recommends microsoft/deberta-xlarge-mnli for better correlation with human scores.
Use it for
- Evaluate machine translation output by scoring candidate translations against reference translations.
- Assess abstractive summarization quality by comparing generated summaries to reference summaries.
- Score image captioning or visual question-answering outputs against reference captions or answers.
- Benchmark text generation models during development without manual human evaluation.
- Compare multiple candidate outputs for the same input and select the highest-scoring one.
Worth the install?
AI-flagged interpretation of the facts on this page. Verify before relying on it.
Yes, with conditions.
BERTScore is well-established with 1913 GitHub stars and solves a real problem in NLP evaluation. However, the package is dormant (last release 2023-02-20) and computationally heavy—it requires PyTorch and GPU access for practical use, and first-run model downloads are slow. Install it if you need embedding-based text evaluation and can afford the compute cost; avoid it if you need active maintenance or lightweight CPU-only evaluation.
Install
bert-score on PyPI
Before you install
Low install friction with a pure Python wheel. The package is dormant (last release 2023-02-20, 1271 days ago) but has 1913 GitHub stars and remains compatible with current transformers versions as of 0.3.13. Eight runtime dependencies including torch and transformers are substantial but standard for NLP work.
Requires PyTorch >= 1.0.0 and Python >= 3.6. First run downloads a pre-trained BERT model (e.g., roberta-large by default), which is computationally intensive and requires GPU for practical use.
License in practice
MIT license is permissive, allowing commercial and private use with minimal restrictions—suitable for most research and production contexts.
Quickstart
pip install bert-score
from bert_score import score
cands = ["the cat is on the mat"]
refs = [["a cat is on the mat"]]
P, R, F1 = score(cands, refs, lang="en")Verify before relying
- Whether the package's performance correlations with human evaluation remain current given the 2023 release date and evolving model landscape.
- Practical memory and compute requirements for scoring at scale with different model choices.
- Current monthly download volume and active user base given dormant maintenance status.
Package facts
| License | MIT permissive |
| Python support | Supports the current Python release >=3.6 |
| Install friction | Low. Pure-Python wheel |
| Runtime dependencies | 8 packagestorchpandastransformersnumpyrequeststqdmmatplotlibpackaging |
| Maintenance | Dormant 1,271 days since the last release |
| Last repo commit | |
| First released | |
| Downloads | 673,466 / month, #5,396 on PyPI 30-day window, as of 2026-08-14 |
| Known vulnerabilities | None known OSV.dev, checked 2026-08-14 |
| Classifiers | Intended Audience :: Science/ResearchLicense :: OSI Approved :: MIT LicenseProgramming Language :: Python :: 3Topic :: Scientific/Engineering :: Artificial Intelligence |
Evidence: bert_score-0.3.13-py3-none-any.whl
Tags
Let your AI agent find packages like this
Example. Real query, live index.
You found this page by searching. An agent finds it by wishing: SkillFed indexes 14,416 PyPI packages by what they can do, searchable in plain language.
wish › “text generation evaluation metric”
- bert-scoreBERTScore computes automatic evaluation metrics for text generation…
- rougeComputes ROUGE scores (Recall-Oriented Understudy for Gisting…
- rouge-metricComputes ROUGE metrics (ROUGE-N, ROUGE-L, ROUGE-W, ROUGE-S, ROUGE-SU)…
Give your agent the search over MCP, or paste the wish link into any chat.
More Artificial Intelligence packages
LiteLLM provides a unified Python interface to call 100+ LLM providers (OpenAI, Anthropic, Gemini, Bedrock, Azure, and others) using OpenAI-compatible API format, available as both a Python SDK and a self-hosted AI Gateway proxy server.
Install it if you need to work with multiple LLM providers or want to centralize LLM routing in your organization.
Client library and CLI tool for downloading, uploading, and managing models, datasets, and repositories on the Hugging Face Hub platform.
Install it if you work with Hugging Face Hub models or datasets.
LangChain provides a framework for building agents and LLM-powered applications by composing language models, tools, and memory through a unified API that abstracts over multiple model providers.
hf-xet provides chunk-based deduplication and efficient file transfer for the Hugging Face Hub, enabling faster uploads and downloads of large files with local disk caching.
Tokenizers converts raw text into token sequences for NLP models, with support for training custom vocabularies and using pre-built tokenizers (BPE, WordPiece) optimized for speed via Rust.
Transformers provides a unified framework for loading, fine-tuning, and running state-of-the-art pretrained models across text, vision, audio, video, and multimodal tasks using PyTorch, JAX, or TensorFlow.
Install it if you need to run or train any transformer-based model for NLP, vision, audio, or multimodal tasks.
See also sentence-transformers · rouge-chinese · unbabel-comet · rouge · rouge-score · bertopic · pytorch-pretrained-bert · keybert · rouge-metric · dtlpymetrics