pytrec-eval

What it is and what it does

pytrec_eval is a Python wrapper around TREC's trec_eval tool, a standard reference implementation for computing Information Retrieval evaluation metrics. It lets you compute measures like MAP (Mean Average Precision) and NDCG (Normalized Discounted Cumulative Gain) by passing relevance judgments and ranked results as Python dictionaries, then get back computed scores for each query.

The package is designed to eliminate the need for custom metric implementations in Python research code. It wraps the compiled trec_eval engine via C++ bindings, so it inherits the correctness and performance of the reference implementation. Installation requires a C++ compiler and Python development headers; the package has been dormant since 2020 but carries no known security issues.

Use it for:

• Evaluate information retrieval systems in Python research pipelines without reimplementing standard metrics.
• Compute ranking quality metrics (MAP, NDCG, etc.) for search or recommendation system experiments.
• Perform statistical significance testing between two ranked result sets using standard IR measures.
• Validate IR models by scoring ranked results against ground-truth relevance judgments in batch.

Worth the install?

AI-flagged interpretation of the facts on this page — verify before relying

Provides Python bindings to compute standard Information Retrieval evaluation measures (MAP, NDCG, and others) against relevance judgments, wrapping the TREC trec_eval tool.

Yes, if you need standard IR evaluation metrics in Python and can meet the C++ build requirements. The package is dormant but stable, carries no known vulnerabilities, and wraps a trusted reference implementation. Install only if you're comfortable with a package that hasn't been updated since 2020 and verify compatibility with your Python version before relying on it in production.

Install

pytrec-eval on PyPI

pip install pytrec-eval

uv add pytrec-eval

poetry add pytrec-eval

Installing pytrec-eval

Before you install

Installation requires a C++ compiler and Python development headers; the package is dormant (last release 2020-09-07, last commit 2023-10-10) but carries no known vulnerabilities and has accumulated 350 repository stars.

License in practice

Licensed under the MIT license (permissive), allowing commercial and private use; note that the underlying trec_eval tool is licensed separately.

Quickstart

pip install pytrec_eval

import pytrec_eval

qrel = {'q1': {'d1': 0, 'd2': 1}}
run = {'q1': {'d1': 1.0, 'd2': 0.0}}
evaluator = pytrec_eval.RelevanceEvaluator(qrel, {'map', 'ndcg'})
results = evaluator.evaluate(run)

Requires a C++ compiler and Python development headers; developed for Python 3.5 and requires Python >= 3.

Verify before relying

• Whether numpy and scipy are runtime dependencies or only build-time requirements (description lists them but fact sheet shows n_runtime: 0).
• Current compatibility with modern Python versions beyond the 3.5 development baseline.
• Whether the package is actively maintained or if dormancy signals end-of-life.

Package facts

Evidence: pytrec_eval-0.5.tar.gz

Keywords: trec_eval, information retrieval, evaluation, ranking

Tags

More General packages