--- id: bert-score version: "0.3.13" license: MIT license_treatment: permissive maintenance: dormant --- # bert-score — PyTorch implementation of BERT score License: permissive · Maintenance: dormant · Downloads: 673.5K/mo ## What it is and what it does BERTScore is an automatic evaluation metric for text generation tasks that uses contextual embeddings from pre-trained BERT models to compare candidate and reference sentences. Rather than relying on surface-level token overlap, it matches words by cosine similarity in embedding space, producing precision, recall, and F1 scores that correlate better with human judgment than traditional metrics. The package wraps torch, transformers, and related dependencies to provide both a simple `score()` function and a `BERTScorer` object that caches models across multiple evaluations. It supports about 130 pre-trained models including BERT variants, RoBERTa, DeBERTa, T5, BART, and multilingual options. The default model is roberta-large, though the documentation recommends microsoft/deberta-xlarge-mnli for better correlation with human scores. Use it for: - Evaluate machine translation output by scoring candidate translations against reference translations. - Assess abstractive summarization quality by comparing generated summaries to reference summaries. - Score image captioning or visual question-answering outputs against reference captions or answers. - Benchmark text generation models during development without manual human evaluation. - Compare multiple candidate outputs for the same input and select the highest-scoring one. ## Worth the install? AI-flagged interpretation of the facts above — verify before relying. BERTScore computes automatic evaluation metrics for text generation by comparing candidate and reference sentences using contextual embeddings from BERT, producing precision, recall, and F1 scores. Yes, with conditions. BERTScore is well-established with 1913 GitHub stars and solves a real problem in NLP evaluation. However, the package is dormant (last release 2023-02-20) and computationally heavy—it requires PyTorch and GPU access for practical use, and first-run model downloads are slow. Install it if you need embedding-based text evaluation and can afford the compute cost; avoid it if you need active maintenance or lightweight CPU-only evaluation. ## Install pip install bert-score uv add bert-score poetry add bert-score ## Installing bert-score Before you install: Low install friction with a pure Python wheel. The package is dormant (last release 2023-02-20, 1271 days ago) but has 1913 GitHub stars and remains compatible with current transformers versions as of 0.3.13. Eight runtime dependencies including torch and transformers are substantial but standard for NLP work. License in practice: MIT license is permissive, allowing commercial and private use with minimal restrictions—suitable for most research and production contexts. Quickstart: pip install bert-score from bert_score import score cands = ["the cat is on the mat"] refs = [["a cat is on the mat"]] P, R, F1 = score(cands, refs, lang="en") Requires PyTorch >= 1.0.0 and Python >= 3.6. First run downloads a pre-trained BERT model (e.g., roberta-large by default), which is computationally intensive and requires GPU for practical use. Verify before relying: - Whether the package's performance correlations with human evaluation remain current given the 2023 release date and evolving model landscape. - Practical memory and compute requirements for scoring at scale with different model choices. - Current monthly download volume and active user base given dormant maintenance status. ## Package facts - License: MIT (permissive) - Python support: supports_current - Install friction: low - Maintenance: dormant - Downloads: 673.5K/month (top 15,000 on PyPI) - Known vulnerabilities: none known ## Tags text generation evaluation metric, automatic text similarity scoring, BERT-based text comparison, NLP evaluation metric, sentence similarity with embeddings, text quality assessment, machine translation evaluation, nlp-evaluation, text-generation, embedding-based-metrics [View on SkillFed](https://skillfed.io/packages/bert-score) · [View on PyPI](https://pypi.org/project/bert-score/)