openevals

What it is and what it does

OpenEvals is a framework for evaluating LLM application outputs using a variety of evaluation strategies. It centers on LLM-as-judge evaluators, which use another LLM to score outputs against custom or prebuilt prompts, but also includes deterministic evaluators for code quality, exact matching, embedding similarity, and agent trajectory validation. The package integrates with LangChain for model access and LangSmith for logging and tracking evaluation results.

The package is designed as a starting point for building custom evaluations specific to your application. It provides prebuilt prompts for common scenarios (correctness, safety, security, RAG quality, code evaluation) and allows flexible customization of scoring, output schemas, and evaluation criteria. It supports async evaluation, multimodal inputs, and multiturn simulation for testing conversational systems.

Use it for:

• Score LLM outputs for quality dimensions like conciseness, correctness, or safety using an LLM-as-judge with prebuilt or custom prompts.
• Evaluate RAG system components (retrieval relevance, groundedness, helpfulness) to measure retrieval and generation quality.
• Validate structured outputs and tool calls from LLM applications using exact-match or LLM-as-judge evaluation.
• Test code generation outputs by extracting and type-checking generated code with Pyright or Mypy.
• Assess agent behavior by matching execution trajectories against expected tool call sequences.

Worth the install?

AI-flagged interpretation of the facts on this page — verify before relying

OpenEvals provides a framework for writing and running evaluators to assess LLM application outputs, including LLM-as-judge evaluators, code evaluators, and prebuilt prompts for common evaluation scenarios.

Yes. OpenEvals is actively maintained, has low install friction, carries a permissive MIT license, and addresses a real need in LLM application development. It provides both prebuilt evaluation patterns and extensibility for custom evals. The four runtime dependencies are all standard LLM/LangChain ecosystem packages. No security vulnerabilities are known.

Install

openevals on PyPI

pip install openevals

uv add openevals

poetry add openevals

Installing openevals

Before you install

Low install friction with a pure Python wheel and four runtime dependencies (langchain, langchain-openai, langsmith, rich). Package is actively maintained with recent releases.

License in practice

MIT license is permissive, allowing use in commercial and proprietary projects with minimal restrictions.

Quickstart

pip install openevals

from openevals.llm import create_llm_as_judge
from openevals.prompts import CONCISENESS_PROMPT

evaluator = create_llm_as_judge(
    prompt=CONCISENESS_PROMPT,
    model="openai:gpt-5.4",
)
result = evaluator(inputs="How is the weather?", outputs="It is sunny.")

Requires OPENAI_API_KEY environment variable set and Python 3.10 or later.

Verify before relying

• Whether prebuilt prompts cover all common evaluation use cases or if custom prompt writing is frequently required.
• Performance characteristics when evaluating large batches of outputs or with different LLM providers.
• Maturity and stability of the multimodal and sandboxed code evaluation features.

Package facts

Evidence: openevals-0.2.0-py3-none-any.whl

Tags

More Testing packages

Further reading