evalplus

What it is and what it does

EvalPlus is a benchmarking and evaluation framework designed to rigorously test code generated by large language models. It extends two popular code-generation benchmarks—HumanEval and MBPP—with significantly more test cases (80x and 35x respectively) to catch edge cases and fragile implementations that simpler test suites miss. Beyond correctness, it includes EvalPerf, a dataset for measuring the efficiency and performance characteristics of generated code.

The framework supports multiple LLM backends (HuggingFace transformers, vLLM, OpenAI, Anthropic, Google Gemini) and can run end-to-end evaluation pipelines: code generation, post-processing, and test execution. It includes Docker-based execution for safe sandboxing of untrusted generated code and provides detailed ranking and scoring to help identify which models produce more robust, production-ready implementations.

Use it for:

• Benchmark and rank LLM code-generation models on standardized test suites with rigorous correctness criteria.
• Evaluate the efficiency of LLM-generated code via performance-exercising tasks and measure runtime/memory characteristics.
• Compare model robustness before and after using EvalPlus tests to identify fragile implementations that pass simple tests.
• Run safe, isolated evaluation of untrusted model-generated code using Docker sandboxing.
• Integrate code-generation evaluation into CI/CD pipelines for continuous model quality monitoring.

Worth the install?

AI-flagged interpretation of the facts on this page — verify before relying

EvalPlus provides a rigorous evaluation framework for code generated by large language models, with extended test suites (HumanEval+, MBPP+) and performance benchmarking (EvalPerf) to assess both correctness and efficiency.

Yes, if you are evaluating or benchmarking LLM code-generation models. The extended test suites and multi-backend support make it the standard tool for rigorous code-generation assessment. The aging maintenance status (663 days since release) is a minor concern but not a blocker—the repository remains active and the framework is stable. Install friction is low and there are no known vulnerabilities.

Install

evalplus on PyPI

pip install evalplus

uv add evalplus

poetry add evalplus

Installing evalplus

Before you install

Low install friction with a pure-Python wheel. Maintenance status is aging (663 days since last release), though the repository remains active with recent commits and moderate community engagement (1798 stars).

License in practice

Apache-2.0 is permissive; you can use, modify, and distribute this package freely in commercial and private projects with minimal restrictions.

Quickstart

pip install evalplus

# Evaluate a model on HumanEval+
evalplus.evaluate --model "ise-uiuc/Magicoder-S-DS-6.7B" \
                  --dataset humaneval \
                  --backend hf \
                  --greedy

Requires Python >= 3.9. Code execution evaluation may require Docker for safety; EvalPerf (efficiency evaluation) is Unix-only and requires perf event access.

Verify before relying

• Whether all 18 runtime dependencies are required for basic evaluation or only for specific backends (vllm, openai, anthropic, etc.)
• Current status of the leaderboard and whether benchmark data is actively maintained post-v0.3.1
• Compatibility of tree-sitter-python with recent Python versions and whether it adds significant install complexity

Package facts

Evidence: evalplus-0.3.1-py3-none-any.whl

Tags

More Testing packages

Further reading