harbor

What it is and what it does

Harbor is an evaluation and optimization framework designed to run agents and language models against standardized benchmarks in isolated sandbox environments. It provides a CLI-driven interface to execute benchmarks like Terminal-Bench-2.0, SWE-Bench, and Aider Polyglot against agents such as Claude Code and OpenHands, with the ability to scale from local Docker execution to distributed cloud providers like Daytona, Modal, and LangSmith.

The framework handles the orchestration of parallel test execution, environment provisioning, and result collection across multiple providers. It integrates with litellm for model abstraction, FastAPI for serving, and various cloud SDKs for distributed execution. Harbor is the official harness for Terminal-Bench-2.0 and is built by the creators of that benchmark, making it purpose-built for systematic agent and model evaluation at scale.

Use it for:

• Run Terminal-Bench-2.0 or other standard benchmarks against Claude Code or other agents locally or on cloud infrastructure.
• Evaluate multiple language models on the same benchmark dataset to compare performance and identify the best fit.
• Scale benchmark execution from a few concurrent runs to hundreds across distributed cloud providers.
• Generate rollouts for reinforcement learning optimization of agents and models.
• Build and share custom benchmarks and evaluation environments with the Harbor framework.

Worth the install?

AI-flagged interpretation of the facts on this page — verify before relying

Harbor is a framework for running and evaluating agents and language models against benchmarks in sandboxed environments, with support for parallel execution across multiple cloud providers.

Yes, if you need to evaluate or benchmark agents and language models systematically. Harbor is actively maintained, has low install friction, and is purpose-built for this use case with support for multiple providers and benchmarks. The Apache-2.0 license is permissive. Requires Python 3.12 and API keys for the services you want to evaluate, plus Docker for local runs. Not relevant if you only need to run individual model inference without structured benchmarking.

Install

harbor on PyPI

pip install harbor

uv add harbor

poetry add harbor

Installing harbor

Before you install

Low install friction with a pure-Python wheel. Active maintenance with a release 4 days ago. Requires Python 3.12 or later. Pulls in 21 runtime dependencies including web frameworks (FastAPI, uvicorn), LLM integration (litellm), and cloud provider SDKs (supabase), which is typical for a multi-provider orchestration tool but adds some dependency weight.

License in practice

Licensed under Apache-2.0 (permissive), allowing free use, modification, and distribution with minimal restrictions—suitable for both open-source and commercial projects.

Quickstart

pip install harbor

export ANTHROPIC_API_KEY=<YOUR-KEY>
harbor run --dataset terminal-bench@2.0 \
  --agent claude-code \
  --model anthropic/claude-opus-4-1 \
  --n-concurrent 4

Requires Python 3.12 or later. Needs API keys for the agent and model provider (e.g., ANTHROPIC_API_KEY for Claude). Docker is required for local benchmark execution.

Verify before relying

• Whether Harbor can be used programmatically via Python imports or is CLI-only.
• What specific agents and models are currently supported beyond the examples shown.
• Whether the framework handles result aggregation and reporting automatically.
• Performance characteristics when running thousands of concurrent environments.

Package facts

Evidence: harbor-0.21.0-py3-none-any.whl

Tags

More Artificial Intelligence packages

Further reading