rubric

What it is and what it does

Rubric is a Python library that automates evaluation of LLM outputs by scoring them against weighted criteria using an LLM as the judge. You define a rubric as a list of requirements, each with a weight; the library calls an LLM (via a pluggable generate function) to evaluate whether each criterion is met, then aggregates those verdicts into a final score. The score can be normalized to 0–1 or returned as a raw weighted sum.

The library ships with three evaluation strategies: PerCriterionGrader makes one inference call per criterion in parallel; PerCriterionOneShotGrader makes a single call for all criteria; and RubricAsJudgeGrader asks the LLM for a holistic 0–100 score. You can use the built-in Gemini integration for quick testing or plug in any LLM provider (OpenAI, Anthropic, local models) by writing a custom async generate function. All grading is async and uses Pydantic for structured outputs.

Use it for

• Evaluate chatbot or RAG system responses against domain-specific quality criteria with weighted importance.
• Detect and penalize specific errors in LLM outputs (e.g., hallucinations, missing facts) using negative-weight criteria.
• Compare multiple LLM models or prompts by scoring their outputs on the same rubric and aggregating results.
• Build automated grading pipelines for educational or content-generation workflows with custom scoring logic.
• Validate structured outputs from LLMs by checking for required fields or format compliance as pass/fail criteria.

Worth the install?

AI-flagged interpretation of the facts on this page. Verify before relying on it.

Install

rubric on PyPI

Before you install

Low install friction with a pure-Python wheel and five runtime dependencies (pydantic, google-genai, tenacity, python-dotenv, pyyaml). Last commit 2026-02-03 and 70 repository stars suggest active maintenance, though the package is classified as Alpha and only 205 days old.

Requires GEMINI_API_KEY environment variable for the default generate function; custom LLM providers (OpenAI, Anthropic, etc.) require their own API keys and custom generate_fn implementation.

License in practice

MIT license permits commercial and private use with minimal restrictions—you may use, modify, and distribute rubric freely as long as you include the license notice.

Quickstart

import asyncio
from rubric import Rubric, default_per_criterion_generate_fn
from rubric.autograders import PerCriterionGrader

async def main():
    rubric = Rubric.from_dict([
        {"weight": 10.0, "requirement": "Response mentions Paris"},
        {"weight": 5.0, "requirement": "Response is concise"}
    ])
    grader = PerCriterionGrader(generate_fn=default_per_criterion_generate_fn)
    result = await rubric.grade("Paris is the capital of France.", autograder=grader)
    print(f"Score: {result.score}")

asyncio.run(main())

Package facts

Evidence: rubric-2.2.0-py3-none-any.whl

Tags

More Python Modules packages

See also openevals · textstat · autoevals · deepeval · ragas · llm · arize-phoenix-evals · instructor · strands-agents-evals