pytrec-eval-terrier

What it is and what it does

pytrec_eval_terrier is a Python wrapper around TREC's trec_eval evaluation tool, designed to eliminate custom implementations of Information Retrieval metrics in Python. It exposes a simple API for computing standard IR measures—such as MAP (Mean Average Precision) and NDCG (Normalized Discounted Cumulative Gain)—on ranked search results against relevance judgments. The package accepts query-document relevance labels and ranked runs as nested dictionaries, then returns computed metrics per query.

This fork, maintained by the University of Glasgow, provides pre-built wheels for modern Python versions (3.10–3.14) and multiple platforms (Linux, macOS, Windows), reducing installation friction compared to building from source. It depends on numpy and scipy for numerical operations and wraps C++ code from the original trec_eval project, making it both fast and reliable for research and production IR evaluation workflows.

Use it for:

• Evaluate search engine or ranking model performance using standard TREC metrics without writing custom metric code.
• Compute statistical significance between two ranked runs to compare IR system improvements.
• Benchmark information retrieval systems in research papers or academic projects.
• Integrate IR evaluation into automated testing or continuous evaluation pipelines.
• Validate ranking quality during development of search applications or recommendation systems.

Worth the install?

AI-flagged interpretation of the facts on this page — verify before relying

Provides Python bindings to TREC's trec_eval tool for computing standard Information Retrieval evaluation measures like MAP and NDCG on ranked search results.

Yes. The package is actively maintained, has no known vulnerabilities, installs cleanly on modern Python versions via pre-built wheels, and solves a real problem—standardized IR evaluation—with a proven, well-cited implementation. Use it if you need to evaluate ranked search results against relevance judgments; the MIT license and permissive treatment pose no barrier.

Install

pytrec-eval-terrier on PyPI

pip install pytrec-eval-terrier

uv add pytrec-eval-terrier

poetry add pytrec-eval-terrier

Installing pytrec-eval-terrier

Before you install

Medium install friction due to compiled C++ components; pre-built wheels available for Python 3.10–3.14 on Linux, macOS, and Windows. Actively maintained with recent releases. Depends on numpy and scipy.

License in practice

Licensed under the MIT license (permissive), allowing broad reuse. Note that the underlying trec_eval tool is licensed separately; check its terms if you modify or redistribute.

Quickstart

pip install pytrec_eval_terrier

import pytrec_eval

qrel = {'q1': {'d1': 0, 'd2': 1}}
run = {'q1': {'d1': 1.0, 'd2': 0.0}}
evaluator = pytrec_eval.RelevanceEvaluator(qrel, {'map', 'ndcg'})
print(evaluator.evaluate(run))

Requires Python 3 or later; numpy and scipy must be installed.

Verify before relying

• Whether the fork maintains full API compatibility with the original pytrec_eval project.
• Performance characteristics when evaluating large-scale ranking runs.
• Availability of documentation beyond the GitHub README and examples.

Package facts

Evidence: pytrec_eval_terrier-0.5.10-cp310-cp310-macosx_10_9_universal2.whl; pytrec_eval_terrier-0.5.10-cp310-cp310-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl; pytrec_eval_terrier-0.5.10-cp310-cp310-musllinux_1_2_x86_64.whl; pytrec_eval_terrier-0.5.10-cp310-cp310-win_amd64.whl; pytrec_eval_terrier-0.5.10-cp311-cp311-macosx_10_9_universal2.whl; pytrec_eval_terrier-0.5.10-cp311-cp311-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl; pytrec_eval_terrier-0.5.10-cp311-cp311-musllinux_1_2_x86_64.whl; pytrec_eval_terrier-0.5.10-cp311-cp311-win_amd64.whl; pytrec_eval_terrier-0.5.10-cp312-cp312-macosx_10_13_universal2.whl; pytrec_eval_terrier-0.5.10-cp312-cp312-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl; pytrec_eval_terrier-0.5.10-cp312-cp312-musllinux_1_2_x86_64.whl; pytrec_eval_terrier-0.5.10-cp312-cp312-win_amd64.whl; pytrec_eval_terrier-0.5.10-cp313-cp313-macosx_10_13_universal2.whl; pytrec_eval_terrier-0.5.10-cp313-cp313-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl; pytrec_eval_terrier-0.5.10-cp313-cp313-musllinux_1_2_x86_64.whl; pytrec_eval_terrier-0.5.10-cp313-cp313-win_amd64.whl; pytrec_eval_terrier-0.5.10-cp314-cp314-macosx_10_15_universal2.whl; pytrec_eval_terrier-0.5.10-cp314-cp314-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl; pytrec_eval_terrier-0.5.10-cp314-cp314-musllinux_1_2_x86_64.whl; pytrec_eval_terrier-0.5.10-cp314-cp314t-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl

Keywords: trec_eval, information retrieval, evaluation, ranking

Tags

More General packages