--- id: inspect-evals version: "0.17.0" license: MIT license_treatment: permissive maintenance: active --- # inspect-evals — Collection of large language model evaluations License: permissive · Maintenance: active · Downloads: 856.9K/mo ## What it is and what it does Inspect Evals is a curated repository of LLM evaluation tasks maintained by the UK AISI, Arcadia Impact, and the Vector Institute. It provides ready-to-run benchmarks for testing language models across multiple providers (OpenAI, Anthropic, Google, Mistral, AWS Bedrock, and others) using the Inspect AI framework. The package bundles evaluation implementations for tasks like ARC and other standardized benchmarks, handling dataset loading, model interaction, and result logging. Developers use Inspect Evals to systematically assess model capabilities without building evaluation infrastructure from scratch. You can run evals via command line or import them as Python objects for programmatic use. The package manages caching of datasets and evaluation artifacts, supports parallel evaluation runs, and integrates with Inspect AI's log viewer for result analysis. Community contributions are accepted through a GitHub-based submission process that validates and registers new evaluations. Use it for: - Run standardized benchmarks like ARC against your LLM to compare performance across model providers - Build a continuous evaluation pipeline by importing eval tasks as Python objects and logging results to track model improvements - Contribute new domain-specific evaluations to the community register by submitting your eval implementation with documentation - Assess AI safety and capability properties using evaluations designed by UK AISI and partner institutions - Compare multiple language models in parallel using eval-set to identify which provider best suits your use case ## Worth the install? AI-flagged interpretation of the facts above — verify before relying. Inspect Evals provides a repository of community-contributed LLM evaluations built on the Inspect AI framework, allowing developers to run standardized benchmarks against language models from multiple providers. Yes. Inspect Evals is actively maintained, has no known vulnerabilities, uses a permissive MIT license, and offers low-friction installation. It's worth installing if you need standardized LLM benchmarking without building evaluation infrastructure yourself. The main constraint is disk space (35–100 GB depending on eval scope) and Python version requirements (3.11–3.12 preferred). Suitable for research, safety assessment, and model selection workflows. ## Install pip install inspect-evals uv add inspect-evals poetry add inspect-evals ## Installing inspect-evals Before you install: Low friction install with a pure Python wheel and 13 runtime dependencies. Active maintenance with a release on 2026-08-14 and 625 repository stars. Requires Python 3.11 or 3.12 for full compatibility; Python 3.13 works for most evals except sciknoweval. License in practice: MIT license permits commercial and private use with minimal restrictions, making the package suitable for both research and production evaluation pipelines. Quickstart: pip install inspect-evals from inspect_ai import eval from inspect_evals.arc import arc_easy eval(arc_easy) Requires Python 3.11 or 3.12; some evaluations need extra dependencies or disk space (35 GB minimum recommended, up to 100 GB for Docker-based evals). API keys for model providers required to run evaluations. Verify before relying: - Whether all 13 runtime dependencies are strictly required or if some are optional for specific evals only - Performance characteristics when running multiple evaluations concurrently with eval-set - Compatibility status with Python 3.13 for the full eval suite beyond the sciknoweval exception ## Package facts - License: MIT (permissive) - Python support: supports_current - Install friction: low - Maintenance: active - Downloads: 856.9K/month (top 5,000 on PyPI) - Known vulnerabilities: none known ## Tags llm evaluation framework, ai model benchmarking, inspect ai evaluations, language model testing, benchmark suite for llms, ai safety evaluation, model capability assessment, llm-benchmarking, ai-evaluation, model-testing [View on SkillFed](https://skillfed.io/packages/inspect-evals) · [View on PyPI](https://pypi.org/project/inspect-evals/)