{"categories":[{"label":"Testing","url":"https://skillfed.io/packages/category/software-development-testing/2"}],"enrichment":{"capability":"OpenEvals provides a framework for writing and running evaluators to assess LLM application outputs, including LLM-as-judge evaluators, code evaluators, and prebuilt prompts for common evaluation scenarios.","skillfed_tags":["llm-evaluation","quality-assurance","langchain-integration"],"use_cases":["Score LLM outputs for quality dimensions like conciseness, correctness, or safety using an LLM-as-judge with prebuilt or custom prompts.","Evaluate RAG system components (retrieval relevance, groundedness, helpfulness) to measure retrieval and generation quality.","Validate structured outputs and tool calls from LLM applications using exact-match or LLM-as-judge evaluation.","Test code generation outputs by extracting and type-checking generated code with Pyright or Mypy.","Assess agent behavior by matching execution trajectories against expected tool call sequences."],"what_it_does":"OpenEvals is a framework for evaluating LLM application outputs using a variety of evaluation strategies. It centers on LLM-as-judge evaluators, which use another LLM to score outputs against custom or prebuilt prompts, but also includes deterministic evaluators for code quality, exact matching, embedding similarity, and agent trajectory validation. The package integrates with LangChain for model access and LangSmith for logging and tracking evaluation results.\n\nThe package is designed as a starting point for building custom evaluations specific to your application. It provides prebuilt prompts for common scenarios (correctness, safety, security, RAG quality, code evaluation) and allows flexible customization of scoring, output schemas, and evaluation criteria. It supports async evaluation, multimodal inputs, and multiturn simulation for testing conversational systems.","worth_installing":"Yes. OpenEvals is actively maintained, has low install friction, carries a permissive MIT license, and addresses a real need in LLM application development. It provides both prebuilt evaluation patterns and extensibility for custom evals. The four runtime dependencies are all standard LLM/LangChain ecosystem packages. No security vulnerabilities are known."},"id":"openevals","links":{"html":"https://skillfed.io/packages/openevals","md":"https://skillfed.io/packages/openevals.md","pypi":"https://pypi.org/project/openevals/"},"maintenance":{"status":"active"},"meta":{"latest_release":"2026-04-07","license_spdx":null,"license_treatment":"permissive","name":"openevals","python_support":"supports_current","summary":"Open-source evaluators for LLM applications"},"popularity":{"monthly_downloads":1251240,"position":4160,"tier":"top_5000"},"security":{"n_vulnerabilities":0},"version":"0.2.0"}