ir-datasets

What it is and what it does

ir_datasets is a Python package that centralizes access to information retrieval benchmarks and training datasets used in academic research and industry. It abstracts away the complexity of locating, downloading, and parsing datasets from multiple sources—handling format differences, encoding issues, and file extraction automatically. The package provides both a Python API and command-line interface, allowing you to iterate over documents, queries, and relevance judgments in a consistent format.

The package handles datasets ranging from small benchmarks to massive collections like ClueWeb (approximately 1 billion documents) by using iterators that avoid loading entire datasets into memory. It also provides fast random-access lookups through document stores, supports slicing operations for parallel processing, and automatically fixes known data quality issues in popular datasets like MS-MARCO. When datasets are not publicly available, it provides instructions on how to obtain them.

Use it for:

• Build and evaluate information retrieval models using standard benchmarks like TREC, MS-MARCO, or BEIR without manually downloading and parsing each dataset.
• Process large document collections in parallel by using efficient iter slicing to partition data across multiple devices or workers.
• Quickly prototype ranking algorithms by accessing queries, documents, and relevance judgments through a unified iterator interface.
• Access multilingual IR datasets (e.g., HC4, CLIRMatrix) for cross-lingual retrieval research without managing separate download and format conversion steps.
• Build dataset-agnostic evaluation pipelines that work across multiple benchmarks by relying on the common interface and automatic data normalization.

Worth the install?

AI-flagged interpretation of the facts on this page — verify before relying

Provides a unified Python interface to download, parse, and iterate over information retrieval benchmarks and training datasets (documents, queries, relevance judgments) from many public sources.

Yes. ir_datasets is actively maintained, has no known vulnerabilities, and solves a real problem for IR researchers and practitioners—centralizing access to fragmented benchmark datasets. The low install friction and permissive MIT license make adoption straightforward. Install it if you work with information retrieval datasets or need to prototype ranking models against standard benchmarks.

Install

ir-datasets on PyPI

pip install ir-datasets

uv add ir-datasets

poetry add ir-datasets

Installing ir-datasets

Before you install

Low install friction with six common runtime dependencies. Actively maintained with a release within the last month and no known vulnerabilities.

License in practice

Licensed under MIT (permissive), allowing free use, modification, and distribution with minimal restrictions.

Quickstart

pip install ir_datasets

import ir_datasets
dataset = ir_datasets.load('msmarco-passage/train')
for doc in dataset.docs_iter():
    print(doc.doc_id, doc.text)

Requires Python 3.8 or later. First use of a dataset will download and cache source files, which may take time and disk space depending on dataset size.

Verify before relying

• Whether the package handles automatic retries or resumable downloads for large dataset transfers
• Memory requirements and performance characteristics for the largest datasets mentioned (e.g., ClueWeb with ~1B documents)

Package facts

Evidence: ir_datasets-0.6.3-py3-none-any.whl

Tags

More Text Processing packages