harbor-rewardkit

What it is and what it does

Harbor Rewardkit is a lightweight grading toolkit for evaluating environment-based tasks. It lets you define verifiers using three approaches: simple programmatic criteria (file checks), LLM judges that score outputs using Claude or other models, and agent judges that can call MCP server tools during evaluation. The package integrates with the Harbor task framework but can be used independently.

You define criteria in TOML files or Python code, then run the verifier against task outputs. It handles LLM model selection, tool access control via allowed_tools lists, and binary or scored evaluation results. The single runtime dependency is litellm, which abstracts LLM provider APIs.

Use it for:

• Score code quality or correctness by having Claude evaluate generated source files against criteria.
• Verify task outputs with simple file-based checks (existence, content matching) without LLM overhead.
• Build agent judges that navigate web pages or interact with systems via MCP tools to validate rendered output.
• Grade benchmark submissions across multiple criteria in a CI/CD pipeline.
• Evaluate multi-step task completion by combining programmatic checks with LLM judgment.

Worth the install?

AI-flagged interpretation of the facts on this page — verify before relying

Harbor Rewardkit defines and runs verifiers for task evaluation, supporting programmatic criteria, LLM judges, and agent judges with MCP server integration.

Yes, if you need lightweight task evaluation with LLM or agent judges. The package is actively maintained, has low install friction, and offers a clean API for both simple checks and complex agent-based grading. Best suited for Harbor framework users or teams building evaluation pipelines for AI-generated outputs.

Install

harbor-rewardkit on PyPI

pip install harbor-rewardkit

uv add harbor-rewardkit

poetry add harbor-rewardkit

Installing harbor-rewardkit

Before you install

Low friction: pure Python wheel with a single runtime dependency (litellm). Active maintenance with recent releases; last commit 2026-08-13. Supports Python 3.12 and 3.13.

License in practice

Apache-2.0 permissive license allows commercial and private use with attribution; no copyleft restrictions.

Quickstart

# Install
uv tool install harbor-rewardkit

# Use programmatic criteria
from rewardkit import criteria
criteria.file_exists("output.txt")
criteria.file_contains("output.txt", "hello")

Requires Python 3.12 or later.

Verify before relying

• Whether litellm's dependencies add meaningful install friction or platform-specific requirements beyond what the wheel distribution handles.
• Whether the package works standalone or requires Harbor task format integration for full functionality.

Package facts

Evidence: harbor_rewardkit-0.1.7-py3-none-any.whl

Keywords: grading, evaluation, rewards, llm, agents, benchmarks

Tags

More Testing packages

Further reading