inspect-evals

What it is and what it does

Inspect Evals is a curated repository of LLM evaluation tasks maintained by the UK AISI, Arcadia Impact, and the Vector Institute. It provides ready-to-run benchmarks for testing language models across multiple providers (OpenAI, Anthropic, Google, Mistral, AWS Bedrock, and others) using the Inspect AI framework. The package bundles evaluation implementations for tasks like ARC and other standardized benchmarks, handling dataset loading, model interaction, and result logging.

Developers use Inspect Evals to systematically assess model capabilities without building evaluation infrastructure from scratch. You can run evals via command line or import them as Python objects for programmatic use. The package manages caching of datasets and evaluation artifacts, supports parallel evaluation runs, and integrates with Inspect AI's log viewer for result analysis. Community contributions are accepted through a GitHub-based submission process that validates and registers new evaluations.

Use it for:

• Run standardized benchmarks like ARC against your LLM to compare performance across model providers
• Build a continuous evaluation pipeline by importing eval tasks as Python objects and logging results to track model improvements
• Contribute new domain-specific evaluations to the community register by submitting your eval implementation with documentation
• Assess AI safety and capability properties using evaluations designed by UK AISI and partner institutions
• Compare multiple language models in parallel using eval-set to identify which provider best suits your use case

Worth the install?

AI-flagged interpretation of the facts on this page — verify before relying

Inspect Evals provides a repository of community-contributed LLM evaluations built on the Inspect AI framework, allowing developers to run standardized benchmarks against language models from multiple providers.

Yes. Inspect Evals is actively maintained, has no known vulnerabilities, uses a permissive MIT license, and offers low-friction installation. It's worth installing if you need standardized LLM benchmarking without building evaluation infrastructure yourself. The main constraint is disk space (35–100 GB depending on eval scope) and Python version requirements (3.11–3.12 preferred). Suitable for research, safety assessment, and model selection workflows.

Install

inspect-evals on PyPI

pip install inspect-evals

uv add inspect-evals

poetry add inspect-evals

Installing inspect-evals

Before you install

Low friction install with a pure Python wheel and 13 runtime dependencies. Active maintenance with a release on 2026-08-14 and 625 repository stars. Requires Python 3.11 or 3.12 for full compatibility; Python 3.13 works for most evals except sciknoweval.

License in practice

MIT license permits commercial and private use with minimal restrictions, making the package suitable for both research and production evaluation pipelines.

Quickstart

pip install inspect-evals

from inspect_ai import eval
from inspect_evals.arc import arc_easy
eval(arc_easy)

Requires Python 3.11 or 3.12; some evaluations need extra dependencies or disk space (35 GB minimum recommended, up to 100 GB for Docker-based evals). API keys for model providers required to run evaluations.

Verify before relying

• Whether all 13 runtime dependencies are strictly required or if some are optional for specific evals only
• Performance characteristics when running multiple evaluations concurrently with eval-set
• Compatibility status with Python 3.13 for the full eval suite beyond the sciknoweval exception

Package facts

Evidence: inspect_evals-0.17.0-py3-none-any.whl

Tags

More Artificial Intelligence packages

Further reading