---
id: unitxt
version: "1.26.10"
license: Apache License Version 2.0, January 2004 http://www.apache.org/licenses/ TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION 1. Definitions. "License" shall mean the terms and conditions for… (full text in the JSON record)
license_treatment: permissive
maintenance: active
---

# unitxt — Load any mixture of text to text data in one line of code

License: permissive · Maintenance: active · Downloads: 92.1K/mo

## What it is and what it does

Unitxt is a Python library designed for evaluating AI model performance in a unified, modular way. It provides a large catalog of pre-built benchmarks and datasets that can be loaded and formatted for any model, supporting text, tables, vision, speech, and code evaluation. The library works model-agnostically with HuggingFace, OpenAI, WatsonX, and custom inference engines, and emphasizes reproducibility through shareable, composable components.

You use Unitxt either through its command-line interface for quick benchmark runs or programmatically by loading datasets, defining tasks with metrics, and running inference and evaluation. It handles data preparation, formatting for chat APIs, and metric computation in one workflow, reducing the boilerplate needed to benchmark models at scale.

Use it for:

- Evaluate a custom LLM against standard benchmarks like MMLU Pro or GPQA without writing dataset loaders
- Run multi-task evaluation across different domains in a single command-line call
- Load and format datasets in chat API format for any model without manual preprocessing
- Build reproducible evaluation pipelines with modular, shareable task and template components
- Compare model performance across text, code, and structured data tasks using unified metrics

## Worth the install?

AI-flagged interpretation of the facts above — verify before relying.

Unitxt provides a unified framework for evaluating AI model performance across text, tables, vision, speech, and code using a modular catalog of benchmarks and datasets.

Yes. Unitxt is actively maintained, has no known vulnerabilities, and offers low-friction installation. It solves a real problem—standardizing AI model evaluation across diverse data types and models—with a permissive license suitable for enterprise use. Install it if you need to benchmark models against established datasets or build reproducible evaluation workflows.

## Install

pip install unitxt
uv add unitxt
poetry add unitxt

## Installing unitxt

Before you install: Low friction installation with four runtime dependencies (datasets, evaluate, scipy, diskcache). Actively maintained with a recent release and no known vulnerabilities.

License in practice: Apache License 2.0 permits commercial use, modification, and distribution with attribution; suitable for enterprise adoption.

Quickstart:

pip install unitxt

from unitxt import load_dataset

dataset = load_dataset(
    card="cards.gpqa.diamond",
    split="test",
    format="formats.chat_api",
)

Verify before relying:

- Whether the catalog contains thousands of datasets as implied by 'thousands of datasets' claim in description
- Performance characteristics and memory footprint when evaluating large-scale benchmarks
- Specific model provider integrations beyond HuggingFace, OpenAI, and WatsonX mentioned

## Package facts

- License: Apache License Version 2.0, January 2004 http://www.apache.org/licenses/ TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION 1. Definitions. "License" shall mean the terms and conditions for… (full text in the JSON record) (permissive)
- Python support: supports_current
- Install friction: low
- Maintenance: active
- Downloads: 92.1K/month (top 15,000 on PyPI)
- Known vulnerabilities: none known

## Tags

AI model evaluation framework, benchmark dataset catalog, text-to-text data preparation, LLM performance evaluation, multi-modal AI evaluation, reproducible model testing, enterprise AI benchmarking, model-evaluation, benchmarking, dataset-catalog

[View on SkillFed](https://skillfed.io/packages/unitxt) · [View on PyPI](https://pypi.org/project/unitxt/)