{"categories":[{"label":"Artificial Intelligence","url":"https://skillfed.io/packages/category/scientific-engineering-artificial-intelligence"}],"enrichment":{"capability":"Harbor is a framework for running and evaluating agents and language models against benchmarks in sandboxed environments, with support for parallel execution across multiple cloud providers.","skillfed_tags":["agent-evaluation","benchmark-framework","distributed-testing"],"use_cases":["Run Terminal-Bench-2.0 or other standard benchmarks against Claude Code or other agents locally or on cloud infrastructure.","Evaluate multiple language models on the same benchmark dataset to compare performance and identify the best fit.","Scale benchmark execution from a few concurrent runs to hundreds across distributed cloud providers.","Generate rollouts for reinforcement learning optimization of agents and models.","Build and share custom benchmarks and evaluation environments with the Harbor framework."],"what_it_does":"Harbor is an evaluation and optimization framework designed to run agents and language models against standardized benchmarks in isolated sandbox environments. It provides a CLI-driven interface to execute benchmarks like Terminal-Bench-2.0, SWE-Bench, and Aider Polyglot against agents such as Claude Code and OpenHands, with the ability to scale from local Docker execution to distributed cloud providers like Daytona, Modal, and LangSmith.\n\nThe framework handles the orchestration of parallel test execution, environment provisioning, and result collection across multiple providers. It integrates with litellm for model abstraction, FastAPI for serving, and various cloud SDKs for distributed execution. Harbor is the official harness for Terminal-Bench-2.0 and is built by the creators of that benchmark, making it purpose-built for systematic agent and model evaluation at scale.","worth_installing":"Yes, if you need to evaluate or benchmark agents and language models systematically. Harbor is actively maintained, has low install friction, and is purpose-built for this use case with support for multiple providers and benchmarks. The Apache-2.0 license is permissive. Requires Python 3.12 and API keys for the services you want to evaluate, plus Docker for local runs. Not relevant if you only need to run individual model inference without structured benchmarking."},"id":"harbor","links":{"html":"https://skillfed.io/packages/harbor","md":"https://skillfed.io/packages/harbor.md","pypi":"https://pypi.org/project/harbor/"},"maintenance":{"status":"active"},"meta":{"latest_release":"2026-08-10","license_spdx":"Apache-2.0","license_treatment":"permissive","name":"harbor","python_support":"supports_current","summary":"A framework for evaluating and optimizing agents and models using sandboxed environments."},"popularity":{"monthly_downloads":12315550,"position":1327,"tier":"top_5000"},"security":{"n_vulnerabilities":0},"version":"0.21.0"}