agentevals

What it is and what it does

AgentEvals is a toolkit for evaluating agentic LLM applications by inspecting the sequence of steps (trajectory) an agent takes to solve a problem. It addresses the challenge of understanding how changes to an agent's configuration or tools affect its behavior downstream, which is difficult because LLMs operate as black boxes. The package provides evaluators that compare actual agent trajectories against expected ones using strict, unordered, or subset/superset matching modes, as well as LLM-as-judge evaluators that use an LLM to reason about trajectory quality.

The package depends on openevals for general evaluation utilities and integrates with LangChain chat models and LangSmith for test orchestration. It represents trajectories as lists of OpenAI-format message dicts or LangChain BaseMessage objects, and supports both synchronous and asynchronous evaluation. The quickstart example shows using an LLM-as-judge to evaluate whether an agent's sequence of tool calls and responses logically solves a user query.

Use it for:

• Validate that an agent calls tools in the correct order and with expected arguments before deploying to production
• Compare agent behavior across model versions or configuration changes to detect regressions in reasoning
• Use an LLM to judge whether an agent's intermediate steps are reasonable and efficient for a given task
• Integrate agent evals into a pytest or LangSmith workflow to catch trajectory issues in CI/CD
• Debug why an agent takes unexpected paths by examining and comparing actual vs. reference trajectories

Worth the install?

AI-flagged interpretation of the facts on this page — verify before relying

Provides evaluators and utilities to assess agent trajectories—the intermediate steps LLM-based agents take while solving problems—using both rule-based matching and LLM-as-judge approaches.

Yes, if you are actively building and testing agentic LLM applications and need to validate agent behavior at the trajectory level. The low install friction and permissive license make adoption easy. However, the aging maintenance status (386 days since last release) means you should verify that it remains compatible with your LLM client versions and check whether the repository is still actively maintained before committing to it for critical production workflows.

Install

agentevals on PyPI

pip install agentevals

uv add agentevals

poetry add agentevals

Installing agentevals

Before you install

Low friction installation with a single runtime dependency (openevals). Maintenance status is aging—last release was 386 days ago—so expect infrequent updates and consider checking the repository for active development before relying on it for production evals.

License in practice

MIT license permits commercial and private use with minimal restrictions, making it suitable for most projects without licensing concerns.

Quickstart

pip install agentevals

from agentevals.trajectory.llm import create_trajectory_llm_as_judge, TRAJECTORY_ACCURACY_PROMPT

trajectory_evaluator = create_trajectory_llm_as_judge(
    prompt=TRAJECTORY_ACCURACY_PROMPT,
    model="openai:o3-mini",
)

result = trajectory_evaluator(outputs=[
    {"role": "user", "content": "What is the weather in SF?"},
    {"role": "assistant", "content": "", "tool_calls": [{"function": {"name": "get_weather", "arguments": "{\"city\": \"SF\"}"}}]},
    {"role": "tool", "content": "It's 80 degrees and sunny in SF."},
    {"role": "assistant", "content": "The weather in SF is 80 degrees and sunny."}
])

Requires OPENAI_API_KEY environment variable set and an LLM client (langchain_openai comes by default, or install openai directly).

Verify before relying

• Whether the package actively maintains compatibility with current LangChain and OpenAI API versions
• Performance characteristics when evaluating large agent trajectories or high-volume eval runs
• Availability of community support or whether issues are actively triaged

Package facts

Evidence: agentevals-0.0.9-py3-none-any.whl

Tags

More Testing packages

Further reading