{"categories":[{"label":"Monitoring","url":"https://skillfed.io/packages/category/system-monitoring/4"}],"enrichment":{"capability":"Judgeval is an SDK for tracing and evaluating LLM-powered agents in production, capturing function inputs/outputs and token usage via OpenTelemetry, then running prompt-based judges to score agent behaviors and surface regressions.","skillfed_tags":["llm-observability","agent-evaluation","opentelemetry"],"use_cases":["Detect and triage agent failures in production by running judges on live traffic and surfacing regressions.","Validate agent fixes by replaying judges against historical traces before shipping to production.","Build a searchable record of agent behavior over time to understand patterns and recurring issues.","Instrument custom tools and functions alongside LLM calls to trace end-to-end agent execution.","Query and inspect agent traces from the CLI or MCP server without writing code."],"what_it_does":"Judgeval is a Python SDK that instruments LLM-powered agents with OpenTelemetry-based tracing and evaluation. It captures function inputs, outputs, and token usage automatically via decorators like @Tracer.observe(), then runs prompt-based judges\u2014custom scorers you define\u2014to evaluate agent behaviors at scale. These judges produce structured, scored outputs that accumulate into a searchable record of how your agent behaved over time.\n\nThe package integrates with multiple LLM providers and frameworks out of the box. You can run judges against live production traffic server-side, or replay them on historical traces to validate fixes before shipping. It includes a CLI for managing traces, judges, and behaviors from the terminal, and an MCP server for querying traces and invoking judges from AI assistants or IDEs.","worth_installing":"Yes. Judgeval is actively maintained, has low install friction, carries a permissive Apache-2.0 license, and solves a real problem\u2014observing and improving LLM agents in production. It requires Python >=3.10 and API credentials (JUDGMENT_API_KEY, JUDGMENT_ORG_ID), which implies a hosted backend service. No known vulnerabilities. Install it if you need production tracing and behavior-based evaluation for agents."},"id":"judgeval","links":{"html":"https://skillfed.io/packages/judgeval","md":"https://skillfed.io/packages/judgeval.md","pypi":"https://pypi.org/project/judgeval/"},"maintenance":{"status":"active"},"meta":{"latest_release":"2026-08-03","license_spdx":"Apache-2.0","license_treatment":"permissive","name":"judgeval","python_support":"supports_current","summary":"The open source post-building layer for Agent Behavior Monitoring."},"popularity":{"monthly_downloads":269024,"position":8265,"tier":"top_15000"},"security":{"n_vulnerabilities":0},"version":"1.3.1"}