{"categories":[{"label":"Testing","url":"https://skillfed.io/packages/category/software-development-testing/3"}],"enrichment":{"capability":"Provides evaluators and utilities to assess agent trajectories\u2014the intermediate steps LLM-based agents take while solving problems\u2014using both rule-based matching and LLM-as-judge approaches.","skillfed_tags":["agent-evaluation","llm-testing","trajectory-analysis"],"use_cases":["Validate that an agent calls tools in the correct order and with expected arguments before deploying to production","Compare agent behavior across model versions or configuration changes to detect regressions in reasoning","Use an LLM to judge whether an agent's intermediate steps are reasonable and efficient for a given task","Integrate agent evals into a pytest or LangSmith workflow to catch trajectory issues in CI/CD","Debug why an agent takes unexpected paths by examining and comparing actual vs. reference trajectories"],"what_it_does":"AgentEvals is a toolkit for evaluating agentic LLM applications by inspecting the sequence of steps (trajectory) an agent takes to solve a problem. It addresses the challenge of understanding how changes to an agent's configuration or tools affect its behavior downstream, which is difficult because LLMs operate as black boxes. The package provides evaluators that compare actual agent trajectories against expected ones using strict, unordered, or subset/superset matching modes, as well as LLM-as-judge evaluators that use an LLM to reason about trajectory quality.\n\nThe package depends on openevals for general evaluation utilities and integrates with LangChain chat models and LangSmith for test orchestration. It represents trajectories as lists of OpenAI-format message dicts or LangChain BaseMessage objects, and supports both synchronous and asynchronous evaluation. The quickstart example shows using an LLM-as-judge to evaluate whether an agent's sequence of tool calls and responses logically solves a user query.","worth_installing":"Yes, if you are actively building and testing agentic LLM applications and need to validate agent behavior at the trajectory level. The low install friction and permissive license make adoption easy. However, the aging maintenance status (386 days since last release) means you should verify that it remains compatible with your LLM client versions and check whether the repository is still actively maintained before committing to it for critical production workflows."},"id":"agentevals","links":{"html":"https://skillfed.io/packages/agentevals","md":"https://skillfed.io/packages/agentevals.md","pypi":"https://pypi.org/project/agentevals/"},"maintenance":{"status":"aging"},"meta":{"latest_release":"2025-07-24","license_spdx":null,"license_treatment":"permissive","name":"agentevals","python_support":"supports_current","summary":"Open-source evaluators for LLM agents"},"popularity":{"monthly_downloads":374429,"position":7145,"tier":"top_15000"},"security":{"n_vulnerabilities":0},"version":"0.0.9"}