--- id: openevals version: "0.2.0" license: MIT license_treatment: permissive maintenance: active --- # openevals — Open-source evaluators for LLM applications License: permissive · Maintenance: active · Downloads: 1.3M/mo ## What it is and what it does OpenEvals is a framework for evaluating LLM application outputs using a variety of evaluation strategies. It centers on LLM-as-judge evaluators, which use another LLM to score outputs against custom or prebuilt prompts, but also includes deterministic evaluators for code quality, exact matching, embedding similarity, and agent trajectory validation. The package integrates with LangChain for model access and LangSmith for logging and tracking evaluation results. The package is designed as a starting point for building custom evaluations specific to your application. It provides prebuilt prompts for common scenarios (correctness, safety, security, RAG quality, code evaluation) and allows flexible customization of scoring, output schemas, and evaluation criteria. It supports async evaluation, multimodal inputs, and multiturn simulation for testing conversational systems. Use it for: - Score LLM outputs for quality dimensions like conciseness, correctness, or safety using an LLM-as-judge with prebuilt or custom prompts. - Evaluate RAG system components (retrieval relevance, groundedness, helpfulness) to measure retrieval and generation quality. - Validate structured outputs and tool calls from LLM applications using exact-match or LLM-as-judge evaluation. - Test code generation outputs by extracting and type-checking generated code with Pyright or Mypy. - Assess agent behavior by matching execution trajectories against expected tool call sequences. ## Worth the install? AI-flagged interpretation of the facts above — verify before relying. OpenEvals provides a framework for writing and running evaluators to assess LLM application outputs, including LLM-as-judge evaluators, code evaluators, and prebuilt prompts for common evaluation scenarios. Yes. OpenEvals is actively maintained, has low install friction, carries a permissive MIT license, and addresses a real need in LLM application development. It provides both prebuilt evaluation patterns and extensibility for custom evals. The four runtime dependencies are all standard LLM/LangChain ecosystem packages. No security vulnerabilities are known. ## Install pip install openevals uv add openevals poetry add openevals ## Installing openevals Before you install: Low install friction with a pure Python wheel and four runtime dependencies (langchain, langchain-openai, langsmith, rich). Package is actively maintained with recent releases. License in practice: MIT license is permissive, allowing use in commercial and proprietary projects with minimal restrictions. Quickstart: pip install openevals from openevals.llm import create_llm_as_judge from openevals.prompts import CONCISENESS_PROMPT evaluator = create_llm_as_judge( prompt=CONCISENESS_PROMPT, model="openai:gpt-5.4", ) result = evaluator(inputs="How is the weather?", outputs="It is sunny.") Requires OPENAI_API_KEY environment variable set and Python 3.10 or later. Verify before relying: - Whether prebuilt prompts cover all common evaluation use cases or if custom prompt writing is frequently required. - Performance characteristics when evaluating large batches of outputs or with different LLM providers. - Maturity and stability of the multimodal and sandboxed code evaluation features. ## Package facts - License: MIT (permissive) - Python support: supports_current - Install friction: low - Maintenance: active - Downloads: 1.3M/month (top 5,000 on PyPI) - Known vulnerabilities: none known ## Tags LLM evaluation framework, LLM-as-judge evaluator, LLM application testing, eval framework for language models, LLM output assessment, evaluation prompts for LLM, LLM quality assurance, llm-evaluation, quality-assurance, langchain-integration [View on SkillFed](https://skillfed.io/packages/openevals) · [View on PyPI](https://pypi.org/project/openevals/)