ir-measures

What it is and what it does

ir_measures is a Python library that standardizes how you compute information retrieval evaluation metrics across different underlying tools. Instead of learning multiple APIs or command-line syntaxes, you import measure objects (AP, nDCG, P@10, etc.) and call calc_aggregate() or iter_calc() with your qrels (relevance judgments) and run (ranked results). It accepts input in multiple formats—dicts, pandas DataFrames, namedtuples, or TREC-formatted files—and outputs per-query or aggregated metric values.

The package wraps pytrec-eval-terrier as its computation engine and is maintained by the Terrier Team at Glasgow. It also exposes a command-line interface for batch evaluation and integrates with PyTerrier for experiment workflows. Active maintenance, no known vulnerabilities, and permissive licensing make it a stable choice for IR research and evaluation pipelines.

Use it for:

• Evaluate a search or ranking system against standard TREC benchmarks using nDCG, AP, and precision metrics.
• Batch-compute metrics from TREC-formatted qrels and run files via command line without writing Python.
• Integrate metric computation into PyTerrier experiments to compare retrieval algorithms.
• Load qrels from ir_datasets and compute per-query and aggregate metrics programmatically.
• Prototype custom ranking evaluation with flexible input formats (dict, DataFrame, or iterables).

Worth the install?

AI-flagged interpretation of the facts on this page — verify before relying

Provides a unified Python interface to compute standard information retrieval evaluation metrics (AP, nDCG, P@k, RR, etc.) from qrels and ranked runs, with both programmatic and command-line APIs.

Yes. ir_measures is actively maintained, has no security vulnerabilities, installs with low friction, and provides a clean, unified interface to a core task in IR research and evaluation. Use it if you need to evaluate ranked results against relevance judgments.

Install

ir-measures on PyPI

pip install ir-measures

uv add ir-measures

poetry add ir-measures

Installing ir-measures

Before you install

Low friction install with a single runtime dependency (pytrec-eval-terrier). The package is actively maintained with a recent release and no known vulnerabilities.

License in practice

Licensed under Apache 2.0 (permissive), allowing use in commercial and private projects with minimal restrictions.

Quickstart

pip install ir-measures

import ir_measures
from ir_measures import AP, nDCG, P

qrels = {'Q0': {'D0': 0, 'D1': 1}}
run = {'Q0': {'D0': 1.2, 'D1': 1.0}}

results = ir_measures.calc_aggregate([AP, nDCG, P@10], qrels, run)
print(results)

Requires Python 3.9 or later.

Verify before relying

• Whether pytrec-eval-terrier requires system libraries or compilation at install time.
• Performance characteristics when evaluating large-scale runs (thousands of queries).
• Support for custom or user-defined metrics beyond the built-in set.

Package facts

Evidence: ir_measures-0.4.3-py3-none-any.whl

Tags

More Information Analysis packages