ir-measures
Provides a common interface to many IR measure tools
Decision gist · record as of 2026-08-14
Yes. ir_measures is actively maintained, has no security vulnerabilities, installs with low friction, and provides a clean, unified interface to a core task in IR research and evaluation. Use it if you need to evaluate ranked results against relevance judgments.AI-flagged interpretation of the facts on this page — verify before relying
Before you install
- Requires Python 3.9 or later.
- Low friction install with a single runtime dependency (pytrec-eval-terrier).
- The package is actively maintained with a recent release and no known vulnerabilities.
License · maintenance · safety
permissive license (permissive) — Licensed under Apache 2.0 (permissive), allowing use in commercial and private projects with minimal restrictions.
last release 2025-11-25 (262 days) · last repo commit 2026-02-17 · 102 stars
0 known vulnerabilities (OSV.dev, 2026-08-14) · 162,501 downloads/mo, #10,598 on PyPI
Alternatives
Verify before relying
pip install ir-measures
import ir_measures
from ir_measures import AP, nDCG, P
qrels = {'Q0': {'D0': 0, 'D1': 1}}
run = {'Q0': {'D0': 1.2, 'D1': 1.0}}
results = ir_measures.calc_aggregate([AP, nDCG, P@10], qrels, run)
print(results)- Whether pytrec-eval-terrier requires system libraries or compilation at install time.
- Performance characteristics when evaluating large-scale runs (thousands of queries).
- Support for custom or user-defined metrics beyond the built-in set.
What it is and what it does
ir_measures is a Python library that standardizes how you compute information retrieval evaluation metrics across different underlying tools. Instead of learning multiple APIs or command-line syntaxes, you import measure objects (AP, nDCG, P@10, etc.) and call calc_aggregate() or iter_calc() with your qrels (relevance judgments) and run (ranked results). It accepts input in multiple formats—dicts, pandas DataFrames, namedtuples, or TREC-formatted files—and outputs per-query or aggregated metric values.
The package wraps pytrec-eval-terrier as its computation engine and is maintained by the Terrier Team at Glasgow. It also exposes a command-line interface for batch evaluation and integrates with PyTerrier for experiment workflows. Active maintenance, no known vulnerabilities, and permissive licensing make it a stable choice for IR research and evaluation pipelines.
Use it for
- Evaluate a search or ranking system against standard TREC benchmarks using nDCG, AP, and precision metrics.
- Batch-compute metrics from TREC-formatted qrels and run files via command line without writing Python.
- Integrate metric computation into PyTerrier experiments to compare retrieval algorithms.
- Load qrels from ir_datasets and compute per-query and aggregate metrics programmatically.
- Prototype custom ranking evaluation with flexible input formats (dict, DataFrame, or iterables).
Worth the install?
AI-flagged interpretation of the facts on this page. Verify before relying on it.
Yes.
ir_measures is actively maintained, has no security vulnerabilities, installs with low friction, and provides a clean, unified interface to a core task in IR research and evaluation. Use it if you need to evaluate ranked results against relevance judgments.
Install
ir-measures on PyPI
Before you install
Low friction install with a single runtime dependency (pytrec-eval-terrier). The package is actively maintained with a recent release and no known vulnerabilities.
Requires Python 3.9 or later.
License in practice
Licensed under Apache 2.0 (permissive), allowing use in commercial and private projects with minimal restrictions.
Quickstart
pip install ir-measures
import ir_measures
from ir_measures import AP, nDCG, P
qrels = {'Q0': {'D0': 0, 'D1': 1}}
run = {'Q0': {'D0': 1.2, 'D1': 1.0}}
results = ir_measures.calc_aggregate([AP, nDCG, P@10], qrels, run)
print(results)Verify before relying
- Whether pytrec-eval-terrier requires system libraries or compilation at install time.
- Performance characteristics when evaluating large-scale runs (thousands of queries).
- Support for custom or user-defined metrics beyond the built-in set.
Package facts
| License | permissive license permissive |
| Python support | Supports the current Python release >=3.9 |
| Install friction | Low. Pure-Python wheel |
| Runtime dependencies | 1 packagepytrec-eval-terrier |
| Maintenance | Actively maintained 262 days since the last release |
| Last repo commit | |
| First released | |
| Downloads | 162,501 / month, #10,598 on PyPI 30-day window, as of 2026-08-14 |
| Known vulnerabilities | None known OSV.dev, checked 2026-08-14 |
| Classifiers | License :: OSI Approved :: Apache Software LicenseOperating System :: OS IndependentProgramming Language :: PythonTopic :: Scientific/Engineering :: Information Analysis |
Evidence: ir_measures-0.4.3-py3-none-any.whl
Tags
Let your AI agent find packages like this
Example. Real query, live index.
You found this page by searching. An agent finds it by wishing: SkillFed indexes 14,416 PyPI packages by what they can do, searchable in plain language.
wish › “information retrieval assessment”
- ir-measuresProvides a unified Python interface to compute standard information…
- pytrec-eval-terrierProvides Python bindings to TREC's trec_eval tool for computing…
- mtebMTEB evaluates text and multimodal embeddings against a suite of…
Give your agent the search over MCP, or paste the wish link into any chat.
More Information Analysis packages
A drop-in replacement for Python's standard `re` module that adds advanced regex features like nested sets, fuzzy matching, lookaround in conditionals, and full Unicode case-folding while maintaining backward compatibility.
pyarrow provides Python bindings to Apache Arrow's C++ libraries for efficient columnar data processing, serialization, and interoperability with pandas, NumPy, and other Python ecosystem tools.
NetworkX provides data structures and algorithms for creating, analyzing, and manipulating graphs and networks, supporting everything from simple undirected graphs to complex directed and weighted networks.
Connects Python applications to Snowflake data warehouses using the DB API 2.0 specification, enabling SQL queries, data transfers, and warehouse operations.
ContourPy calculates contours of 2D quadrilateral grids using C++11 algorithms wrapped in Python, offering serial and multithreaded implementations without requiring Matplotlib as a dependency.
Snowpark Python provides APIs to query and process data directly in Snowflake without moving data to your local system, with support for both native Snowpark and pandas-compatible interfaces.
Install it if you use Snowflake and want to process data without moving it to your application layer.
See also ir-datasets · pytrec-eval · ranx · pytrec-eval-terrier · trec-car-tools · mir-eval · rax · image-similarity-measures · rank-bm25 · pyannote-metrics