harbor
A framework for evaluating and optimizing agents and models using sandboxed environments.
Decision gist · record as of 2026-08-14
Yes, if you need to evaluate or benchmark agents and language models systematically. Harbor is actively maintained, has low install friction, and is purpose-built for this use case with support for multiple providers and benchmarks. The Apache-2.0 license is permissive. Requires Python 3.12 and API keys for the services you want to evaluate, plus Docker for local runs. Not relevant if you only need to run individual model inference without structured benchmarking.AI-flagged interpretation of the facts on this page — verify before relying
Before you install
- Requires Python 3.12 or later.
- Needs API keys for the agent and model provider (e.g., ANTHROPIC_API_KEY for Claude).
- Docker is required for local benchmark execution.
License · maintenance · safety
Apache-2.0 (permissive) — Licensed under Apache-2.0 (permissive), allowing free use, modification, and distribution with minimal restrictions—suitable for both open-source and commercial projects.
last release 2026-08-10 (4 days)
0 known vulnerabilities (OSV.dev, 2026-08-14) · 12,315,550 downloads/mo, #1,327 on PyPI
Alternatives
Verify before relying
pip install harbor
export ANTHROPIC_API_KEY=<YOUR-KEY>
harbor run --dataset terminal-bench@2.0 \
--agent claude-code \
--model anthropic/claude-opus-4-1 \
--n-concurrent 4- Whether Harbor can be used programmatically via Python imports or is CLI-only.
- What specific agents and models are currently supported beyond the examples shown.
- Whether the framework handles result aggregation and reporting automatically.
- Performance characteristics when running thousands of concurrent environments.
What it is and what it does
Harbor is an evaluation and optimization framework designed to run agents and language models against standardized benchmarks in isolated sandbox environments. It provides a CLI-driven interface to execute benchmarks like Terminal-Bench-2.0, SWE-Bench, and Aider Polyglot against agents such as Claude Code and OpenHands, with the ability to scale from local Docker execution to distributed cloud providers like Daytona, Modal, and LangSmith.
The framework handles the orchestration of parallel test execution, environment provisioning, and result collection across multiple providers. It integrates with litellm for model abstraction, FastAPI for serving, and various cloud SDKs for distributed execution. Harbor is the official harness for Terminal-Bench-2.0 and is built by the creators of that benchmark, making it purpose-built for systematic agent and model evaluation at scale.
Use it for
- Run Terminal-Bench-2.0 or other standard benchmarks against Claude Code or other agents locally or on cloud infrastructure.
- Evaluate multiple language models on the same benchmark dataset to compare performance and identify the best fit.
- Scale benchmark execution from a few concurrent runs to hundreds across distributed cloud providers.
- Generate rollouts for reinforcement learning optimization of agents and models.
- Build and share custom benchmarks and evaluation environments with the Harbor framework.
Worth the install?
AI-flagged interpretation of the facts on this page. Verify before relying on it.
Yes, if you need to evaluate or benchmark agents and language models systematically.
Harbor is actively maintained, has low install friction, and is purpose-built for this use case with support for multiple providers and benchmarks. The Apache-2.0 license is permissive. Requires Python 3.12 and API keys for the services you want to evaluate, plus Docker for local runs. Not relevant if you only need to run individual model inference without structured benchmarking.
Install
harbor on PyPI
Before you install
Low install friction with a pure-Python wheel. Active maintenance with a release 4 days ago. Requires Python 3.12 or later. Pulls in 21 runtime dependencies including web frameworks (FastAPI, uvicorn), LLM integration (litellm), and cloud provider SDKs (supabase), which is typical for a multi-provider orchestration tool but adds some dependency weight.
Requires Python 3.12 or later. Needs API keys for the agent and model provider (e.g., ANTHROPIC_API_KEY for Claude). Docker is required for local benchmark execution.
License in practice
Licensed under Apache-2.0 (permissive), allowing free use, modification, and distribution with minimal restrictions—suitable for both open-source and commercial projects.
Quickstart
pip install harbor
export ANTHROPIC_API_KEY=<YOUR-KEY>
harbor run --dataset terminal-bench@2.0 \
--agent claude-code \
--model anthropic/claude-opus-4-1 \
--n-concurrent 4Verify before relying
- Whether Harbor can be used programmatically via Python imports or is CLI-only.
- What specific agents and models are currently supported beyond the examples shown.
- Whether the framework handles result aggregation and reporting automatically.
- Performance characteristics when running thousands of concurrent environments.
Package facts
| License | Apache-2.0 permissive |
| Python support | Supports the current Python release >=3.12 |
| Install friction | Low. Pure-Python wheel |
| Runtime dependencies | 21 packagespydanticshortuuidtyperrequestspyyamlrichtomltenacitypython-dotenvlitellmjinja2dirhashpackagingfastapiuvicornpathspecsupabasehttpxfilelockplatformdirspyjwt |
| Maintenance | Actively maintained 4 days since the last release |
| First released | |
| Downloads | 12,315,550 / month, #1,327 on PyPI 30-day window, as of 2026-08-14 |
| Known vulnerabilities | None known OSV.dev, checked 2026-08-14 |
Evidence: harbor-0.21.0-py3-none-any.whl
Tags
Let your AI agent find packages like this
Example. Real query, live index.
You found this page by searching. An agent finds it by wishing: SkillFed indexes 14,416 PyPI packages by what they can do, searchable in plain language.
wish › “benchmark runner parallel”
- harborHarbor is a framework for running and evaluating agents and language…
- rtestA Python test runner built in Rust that uses static AST analysis for…
- concurrencytestRuns unittest test suites concurrently across multiple worker…
Give your agent the search over MCP, or paste the wish link into any chat.
More Artificial Intelligence packages
LiteLLM provides a unified Python interface to call 100+ LLM providers (OpenAI, Anthropic, Gemini, Bedrock, Azure, and others) using OpenAI-compatible API format, available as both a Python SDK and a self-hosted AI Gateway proxy server.
Install it if you need to work with multiple LLM providers or want to centralize LLM routing in your organization.
Client library and CLI tool for downloading, uploading, and managing models, datasets, and repositories on the Hugging Face Hub platform.
Install it if you work with Hugging Face Hub models or datasets.
LangChain provides a framework for building agents and LLM-powered applications by composing language models, tools, and memory through a unified API that abstracts over multiple model providers.
hf-xet provides chunk-based deduplication and efficient file transfer for the Hugging Face Hub, enabling faster uploads and downloads of large files with local disk caching.
Tokenizers converts raw text into token sequences for NLP models, with support for training custom vocabularies and using pre-built tokenizers (BPE, WordPiece) optimized for speed via Rust.
Transformers provides a unified framework for loading, fine-tuning, and running state-of-the-art pretrained models across text, vision, audio, video, and multimodal tasks using PyTorch, JAX, or TensorFlow.
Install it if you need to run or train any transformer-based model for NLP, vision, audio, or multimodal tasks.
See also harbor-rewardkit · harborapi · terminal-bench · pydantic-deep · nemo-evaluator · nemo-gym · swe-rex · deepagents · inspect-evals · omnigent