rubric

What it is and what it does

Rubric is a Python library that automates evaluation of LLM outputs by scoring them against weighted criteria using an LLM as the judge. You define a rubric as a list of requirements, each with a weight; the library calls an LLM (via a pluggable generate function) to evaluate whether each criterion is met, then aggregates those verdicts into a final score. The score can be normalized to 0–1 or returned as a raw weighted sum.

The library ships with three evaluation strategies: PerCriterionGrader makes one inference call per criterion in parallel; PerCriterionOneShotGrader makes a single call for all criteria; and RubricAsJudgeGrader asks the LLM for a holistic 0–100 score. You can use the built-in Gemini integration for quick testing or plug in any LLM provider (OpenAI, Anthropic, local models) by writing a custom async generate function. All grading is async and uses Pydantic for structured outputs.

Use it for:

• Evaluate chatbot or RAG system responses against domain-specific quality criteria with weighted importance.
• Detect and penalize specific errors in LLM outputs (e.g., hallucinations, missing facts) using negative-weight criteria.
• Compare multiple LLM models or prompts by scoring their outputs on the same rubric and aggregating results.
• Build automated grading pipelines for educational or content-generation workflows with custom scoring logic.
• Validate structured outputs from LLMs by checking for required fields or format compliance as pass/fail criteria.

Worth the install?

AI-flagged interpretation of the facts on this page — verify before relying

Rubric is a Python library for evaluating LLM outputs against weighted criteria using structured rubrics, returning normalized or raw scores based on how many criteria are met.

Yes, if you need to evaluate LLM outputs programmatically against weighted criteria. The library is young (Alpha, 205 days old) but actively maintained, has low install friction, and offers a clean abstraction over LLM-based grading with multiple evaluation strategies. Start with the default Gemini integration for prototyping; for production, implement a custom generate_fn with your preferred LLM provider. No known security vulnerabilities.

Install

rubric on PyPI

pip install rubric

uv add rubric

poetry add rubric

Installing rubric

Before you install

Low install friction with a pure-Python wheel and five runtime dependencies (pydantic, google-genai, tenacity, python-dotenv, pyyaml). Last commit 2026-02-03 and 70 repository stars suggest active maintenance, though the package is classified as Alpha and only 205 days old.

License in practice

MIT license permits commercial and private use with minimal restrictions—you may use, modify, and distribute rubric freely as long as you include the license notice.

Quickstart

import asyncio
from rubric import Rubric, default_per_criterion_generate_fn
from rubric.autograders import PerCriterionGrader

async def main():
    rubric = Rubric.from_dict([
        {"weight": 10.0, "requirement": "Response mentions Paris"},
        {"weight": 5.0, "requirement": "Response is concise"}
    ])
    grader = PerCriterionGrader(generate_fn=default_per_criterion_generate_fn)
    result = await rubric.grade("Paris is the capital of France.", autograder=grader)
    print(f"Score: {result.score}")

asyncio.run(main())

Requires GEMINI_API_KEY environment variable for the default generate function; custom LLM providers (OpenAI, Anthropic, etc.) require their own API keys and custom generate_fn implementation.

Verify before relying

• Whether the three autograder strategies (PerCriterionGrader, PerCriterionOneShotGrader, RubricAsJudgeGrader) have documented performance or latency trade-offs.
• Whether custom generate functions can be synchronous or must be async.
• Behavior and scoring when all criteria weights are negative (error-detection rubrics) with edge cases.

Package facts

Evidence: rubric-2.2.0-py3-none-any.whl

Keywords: rubrics, grading, eval, rubric

Tags

More Python Modules packages