{"categories":[{"label":"Testing","url":"https://skillfed.io/packages/category/software-development-testing/5"}],"enrichment":{"capability":"EvalPlus provides a rigorous evaluation framework for code generated by large language models, with extended test suites (HumanEval+, MBPP+) and performance benchmarking (EvalPerf) to assess both correctness and efficiency.","skillfed_tags":["llm-evaluation","code-benchmarking","ml-ops"],"use_cases":["Benchmark and rank LLM code-generation models on standardized test suites with rigorous correctness criteria.","Evaluate the efficiency of LLM-generated code via performance-exercising tasks and measure runtime/memory characteristics.","Compare model robustness before and after using EvalPlus tests to identify fragile implementations that pass simple tests.","Run safe, isolated evaluation of untrusted model-generated code using Docker sandboxing.","Integrate code-generation evaluation into CI/CD pipelines for continuous model quality monitoring."],"what_it_does":"EvalPlus is a benchmarking and evaluation framework designed to rigorously test code generated by large language models. It extends two popular code-generation benchmarks\u2014HumanEval and MBPP\u2014with significantly more test cases (80x and 35x respectively) to catch edge cases and fragile implementations that simpler test suites miss. Beyond correctness, it includes EvalPerf, a dataset for measuring the efficiency and performance characteristics of generated code.\n\nThe framework supports multiple LLM backends (HuggingFace transformers, vLLM, OpenAI, Anthropic, Google Gemini) and can run end-to-end evaluation pipelines: code generation, post-processing, and test execution. It includes Docker-based execution for safe sandboxing of untrusted generated code and provides detailed ranking and scoring to help identify which models produce more robust, production-ready implementations.","worth_installing":"Yes, if you are evaluating or benchmarking LLM code-generation models. The extended test suites and multi-backend support make it the standard tool for rigorous code-generation assessment. The aging maintenance status (663 days since release) is a minor concern but not a blocker\u2014the repository remains active and the framework is stable. Install friction is low and there are no known vulnerabilities."},"id":"evalplus","links":{"html":"https://skillfed.io/packages/evalplus","md":"https://skillfed.io/packages/evalplus.md","pypi":"https://pypi.org/project/evalplus/"},"maintenance":{"status":"aging"},"meta":{"latest_release":"2024-10-20","license_spdx":null,"license_treatment":"permissive","name":"evalplus","python_support":"supports_current","summary":"\"EvalPlus for rigourous evaluation of LLM-synthesized code\""},"popularity":{"monthly_downloads":159707,"position":10690,"tier":"top_15000"},"security":{"n_vulnerabilities":0},"version":"0.3.1"}