judgeval

What it is and what it does

Judgeval is a Python SDK that instruments LLM-powered agents with OpenTelemetry-based tracing and evaluation. It captures function inputs, outputs, and token usage automatically via decorators like @Tracer.observe(), then runs prompt-based judges—custom scorers you define—to evaluate agent behaviors at scale. These judges produce structured, scored outputs that accumulate into a searchable record of how your agent behaved over time.

The package integrates with multiple LLM providers and frameworks out of the box. You can run judges against live production traffic server-side, or replay them on historical traces to validate fixes before shipping. It includes a CLI for managing traces, judges, and behaviors from the terminal, and an MCP server for querying traces and invoking judges from AI assistants or IDEs.

Use it for:

• Detect and triage agent failures in production by running judges on live traffic and surfacing regressions.
• Validate agent fixes by replaying judges against historical traces before shipping to production.
• Build a searchable record of agent behavior over time to understand patterns and recurring issues.
• Instrument custom tools and functions alongside LLM calls to trace end-to-end agent execution.
• Query and inspect agent traces from the CLI or MCP server without writing code.

Worth the install?

AI-flagged interpretation of the facts on this page — verify before relying

Judgeval is an SDK for tracing and evaluating LLM-powered agents in production, capturing function inputs/outputs and token usage via OpenTelemetry, then running prompt-based judges to score agent behaviors and surface regressions.

Yes. Judgeval is actively maintained, has low install friction, carries a permissive Apache-2.0 license, and solves a real problem—observing and improving LLM agents in production. It requires Python >=3.10 and API credentials (JUDGMENT_API_KEY, JUDGMENT_ORG_ID), which implies a hosted backend service. No known vulnerabilities. Install it if you need production tracing and behavior-based evaluation for agents.

Install

judgeval on PyPI

pip install judgeval

uv add judgeval

poetry add judgeval

Installing judgeval

Before you install

Low friction: pure Python wheel with eight runtime dependencies. Active maintenance—last commit 2026-08-10, 1057 GitHub stars, released 2026-08-03.

License in practice

Apache-2.0 (permissive): you may use, modify, and distribute this package freely in commercial and private projects, provided you include the license notice and state significant changes.

Quickstart

pip install judgeval

from judgeval import Tracer

Tracer.init(project_name="my-project")

@Tracer.observe(span_type="agent")
def run_agent(question: str) -> str:
    return "response"

Requires Python >=3.10; JUDGMENT_API_KEY and JUDGMENT_ORG_ID environment variables must be set to send traces to the Judgment backend.

Verify before relying

• Specific latency overhead of server-side scoring on live production traffic.
• Whether Slack alert configuration is built-in or requires external setup.
• Retention and query performance characteristics of the trace storage backend.
• Cost model for the Judgment backend service (API key required suggests hosted service).
• Auto-instrumentation behavior and token usage capture mechanics with specific LLM providers.

Package facts

Evidence: judgeval-1.3.1-py3-none-any.whl

Tags

More Monitoring packages

Further reading