---
id: harbor-rewardkit
version: "0.1.7"
license: Apache-2.0
license_treatment: permissive
maintenance: active
---

# harbor-rewardkit — Lightweight grading toolkit for environment-based tasks.

License: permissive · Maintenance: active · Downloads: 7.7M/mo

## What it is and what it does

Harbor Rewardkit is a lightweight grading toolkit for evaluating environment-based tasks. It lets you define verifiers using three approaches: simple programmatic criteria (file checks), LLM judges that score outputs using Claude or other models, and agent judges that can call MCP server tools during evaluation. The package integrates with the Harbor task framework but can be used independently.

You define criteria in TOML files or Python code, then run the verifier against task outputs. It handles LLM model selection, tool access control via allowed_tools lists, and binary or scored evaluation results. The single runtime dependency is litellm, which abstracts LLM provider APIs.

Use it for:

- Score code quality or correctness by having Claude evaluate generated source files against criteria.
- Verify task outputs with simple file-based checks (existence, content matching) without LLM overhead.
- Build agent judges that navigate web pages or interact with systems via MCP tools to validate rendered output.
- Grade benchmark submissions across multiple criteria in a CI/CD pipeline.
- Evaluate multi-step task completion by combining programmatic checks with LLM judgment.

## Worth the install?

AI-flagged interpretation of the facts above — verify before relying.

Harbor Rewardkit defines and runs verifiers for task evaluation, supporting programmatic criteria, LLM judges, and agent judges with MCP server integration.

Yes, if you need lightweight task evaluation with LLM or agent judges. The package is actively maintained, has low install friction, and offers a clean API for both simple checks and complex agent-based grading. Best suited for Harbor framework users or teams building evaluation pipelines for AI-generated outputs.

## Install

pip install harbor-rewardkit
uv add harbor-rewardkit
poetry add harbor-rewardkit

## Installing harbor-rewardkit

Before you install: Low friction: pure Python wheel with a single runtime dependency (litellm). Active maintenance with recent releases; last commit 2026-08-13. Supports Python 3.12 and 3.13.

License in practice: Apache-2.0 permissive license allows commercial and private use with attribution; no copyleft restrictions.

Quickstart:

# Install
uv tool install harbor-rewardkit

# Use programmatic criteria
from rewardkit import criteria
criteria.file_exists("output.txt")
criteria.file_contains("output.txt", "hello")

Requires Python 3.12 or later.

Verify before relying:

- Whether litellm's dependencies add meaningful install friction or platform-specific requirements beyond what the wheel distribution handles.
- Whether the package works standalone or requires Harbor task format integration for full functionality.

## Package facts

- License: Apache-2.0 (permissive)
- Python support: supports_current
- Install friction: low
- Maintenance: active
- Downloads: 7.7M/month (top 5,000 on PyPI)
- Known vulnerabilities: none known

## Tags

task evaluation framework, llm judge verifier, grading toolkit, automated criteria checking, agent evaluation, benchmark scoring, mcp server integration, llm-evaluation, task-grading, mcp-agents

[View on SkillFed](https://skillfed.io/packages/harbor-rewardkit) · [View on PyPI](https://pypi.org/project/harbor-rewardkit/)