---
id: evalplus
version: "0.3.1"
license: Apache-2.0
license_treatment: permissive
maintenance: aging
---

# evalplus — "EvalPlus for rigourous evaluation of LLM-synthesized code"

License: permissive · Maintenance: aging · Downloads: 159.7K/mo

## What it is and what it does

EvalPlus is a benchmarking and evaluation framework designed to rigorously test code generated by large language models. It extends two popular code-generation benchmarks—HumanEval and MBPP—with significantly more test cases (80x and 35x respectively) to catch edge cases and fragile implementations that simpler test suites miss. Beyond correctness, it includes EvalPerf, a dataset for measuring the efficiency and performance characteristics of generated code.

The framework supports multiple LLM backends (HuggingFace transformers, vLLM, OpenAI, Anthropic, Google Gemini) and can run end-to-end evaluation pipelines: code generation, post-processing, and test execution. It includes Docker-based execution for safe sandboxing of untrusted generated code and provides detailed ranking and scoring to help identify which models produce more robust, production-ready implementations.

Use it for:

- Benchmark and rank LLM code-generation models on standardized test suites with rigorous correctness criteria.
- Evaluate the efficiency of LLM-generated code via performance-exercising tasks and measure runtime/memory characteristics.
- Compare model robustness before and after using EvalPlus tests to identify fragile implementations that pass simple tests.
- Run safe, isolated evaluation of untrusted model-generated code using Docker sandboxing.
- Integrate code-generation evaluation into CI/CD pipelines for continuous model quality monitoring.

## Worth the install?

AI-flagged interpretation of the facts above — verify before relying.

EvalPlus provides a rigorous evaluation framework for code generated by large language models, with extended test suites (HumanEval+, MBPP+) and performance benchmarking (EvalPerf) to assess both correctness and efficiency.

Yes, if you are evaluating or benchmarking LLM code-generation models. The extended test suites and multi-backend support make it the standard tool for rigorous code-generation assessment. The aging maintenance status (663 days since release) is a minor concern but not a blocker—the repository remains active and the framework is stable. Install friction is low and there are no known vulnerabilities.

## Install

pip install evalplus
uv add evalplus
poetry add evalplus

## Installing evalplus

Before you install: Low install friction with a pure-Python wheel. Maintenance status is aging (663 days since last release), though the repository remains active with recent commits and moderate community engagement (1798 stars).

License in practice: Apache-2.0 is permissive; you can use, modify, and distribute this package freely in commercial and private projects with minimal restrictions.

Quickstart:

pip install evalplus

# Evaluate a model on HumanEval+
evalplus.evaluate --model "ise-uiuc/Magicoder-S-DS-6.7B" \
                  --dataset humaneval \
                  --backend hf \
                  --greedy

Requires Python >= 3.9. Code execution evaluation may require Docker for safety; EvalPerf (efficiency evaluation) is Unix-only and requires perf event access.

Verify before relying:

- Whether all 18 runtime dependencies are required for basic evaluation or only for specific backends (vllm, openai, anthropic, etc.)
- Current status of the leaderboard and whether benchmark data is actively maintained post-v0.3.1
- Compatibility of tree-sitter-python with recent Python versions and whether it adds significant install complexity

## Package facts

- License: Apache-2.0 (permissive)
- Python support: supports_current
- Install friction: low
- Maintenance: aging
- Downloads: 159.7K/month (top 15,000 on PyPI)
- Known vulnerabilities: none known

## Tags

llm code generation evaluation, humaneval+ mbpp+ benchmark, code correctness testing framework, llm4code evaluation harness, code efficiency performance testing, llm-evaluation, code-benchmarking, ml-ops

[View on SkillFed](https://skillfed.io/packages/evalplus) · [View on PyPI](https://pypi.org/project/evalplus/)