---
id: harbor
version: "0.21.0"
license: Apache-2.0
license_treatment: permissive
maintenance: active
---

# harbor — A framework for evaluating and optimizing agents and models using sandboxed environments.

License: permissive · Maintenance: active · Downloads: 12.3M/mo

## What it is and what it does

Harbor is an evaluation and optimization framework designed to run agents and language models against standardized benchmarks in isolated sandbox environments. It provides a CLI-driven interface to execute benchmarks like Terminal-Bench-2.0, SWE-Bench, and Aider Polyglot against agents such as Claude Code and OpenHands, with the ability to scale from local Docker execution to distributed cloud providers like Daytona, Modal, and LangSmith.

The framework handles the orchestration of parallel test execution, environment provisioning, and result collection across multiple providers. It integrates with litellm for model abstraction, FastAPI for serving, and various cloud SDKs for distributed execution. Harbor is the official harness for Terminal-Bench-2.0 and is built by the creators of that benchmark, making it purpose-built for systematic agent and model evaluation at scale.

Use it for:

- Run Terminal-Bench-2.0 or other standard benchmarks against Claude Code or other agents locally or on cloud infrastructure.
- Evaluate multiple language models on the same benchmark dataset to compare performance and identify the best fit.
- Scale benchmark execution from a few concurrent runs to hundreds across distributed cloud providers.
- Generate rollouts for reinforcement learning optimization of agents and models.
- Build and share custom benchmarks and evaluation environments with the Harbor framework.

## Worth the install?

AI-flagged interpretation of the facts above — verify before relying.

Harbor is a framework for running and evaluating agents and language models against benchmarks in sandboxed environments, with support for parallel execution across multiple cloud providers.

Yes, if you need to evaluate or benchmark agents and language models systematically. Harbor is actively maintained, has low install friction, and is purpose-built for this use case with support for multiple providers and benchmarks. The Apache-2.0 license is permissive. Requires Python 3.12 and API keys for the services you want to evaluate, plus Docker for local runs. Not relevant if you only need to run individual model inference without structured benchmarking.

## Install

pip install harbor
uv add harbor
poetry add harbor

## Installing harbor

Before you install: Low install friction with a pure-Python wheel. Active maintenance with a release 4 days ago. Requires Python 3.12 or later. Pulls in 21 runtime dependencies including web frameworks (FastAPI, uvicorn), LLM integration (litellm), and cloud provider SDKs (supabase), which is typical for a multi-provider orchestration tool but adds some dependency weight.

License in practice: Licensed under Apache-2.0 (permissive), allowing free use, modification, and distribution with minimal restrictions—suitable for both open-source and commercial projects.

Quickstart:

pip install harbor

export ANTHROPIC_API_KEY=<YOUR-KEY>
harbor run --dataset terminal-bench@2.0 \
  --agent claude-code \
  --model anthropic/claude-opus-4-1 \
  --n-concurrent 4

Requires Python 3.12 or later. Needs API keys for the agent and model provider (e.g., ANTHROPIC_API_KEY for Claude). Docker is required for local benchmark execution.

Verify before relying:

- Whether Harbor can be used programmatically via Python imports or is CLI-only.
- What specific agents and models are currently supported beyond the examples shown.
- Whether the framework handles result aggregation and reporting automatically.
- Performance characteristics when running thousands of concurrent environments.

## Package facts

- License: Apache-2.0 (permissive)
- Python support: supports_current
- Install friction: low
- Maintenance: active
- Downloads: 12.3M/month (top 5,000 on PyPI)
- Known vulnerabilities: none known

## Tags

agent evaluation framework, language model benchmarking, distributed sandbox testing, LLM agent harness, benchmark runner parallel, cloud-based model evaluation, agent performance testing, agent-evaluation, benchmark-framework, distributed-testing

[View on SkillFed](https://skillfed.io/packages/harbor) · [View on PyPI](https://pypi.org/project/harbor/)