terminal-bench

What it is and what it does

Terminal-Bench is a benchmarking framework designed to evaluate how well AI agents can autonomously complete real-world terminal tasks. It consists of a dataset of approximately 100 tasks (each with an English instruction, test script, and reference solution) and an execution harness that connects language models to a sandboxed terminal environment. The harness supports multiple LLM providers via integrations with anthropic, openai, and litellm, and can run tasks concurrently across Docker containers.

The package is aimed at researchers and developers building LLM agents, benchmarking frameworks, or stress-testing system-level reasoning capabilities. It provides a reproducible, practical evaluation suite for terminal-based workflows—tasks range from compiling code to training models to setting up servers. The CLI tool `tb` lets you run evaluations, submit results to a leaderboard, and contribute new tasks or adapters to the community.

Use it for:

• Evaluate an LLM agent's ability to autonomously complete real-world terminal tasks and compare performance across models
• Benchmark a custom AI agent framework against a standardized task suite with reproducible results
• Stress-test system-level reasoning in language models by running end-to-end workflows like server setup or model training
• Contribute new terminal-based tasks to the Terminal-Bench dataset to expand the evaluation suite
• Submit agent evaluation results to the Terminal-Bench leaderboard for public comparison

Worth the install?

AI-flagged interpretation of the facts on this page — verify before relying

Terminal-Bench provides a benchmark suite and execution harness for evaluating AI agents' ability to complete real-world terminal tasks autonomously, from code compilation to server setup.

Yes, with conditions. Install if you are actively developing or evaluating AI agents for terminal tasks and can meet the Python 3.12+ and Docker requirements. The package has low install friction and no known vulnerabilities, but verify the license terms before use in commercial contexts. The aging maintenance status and unclear license are minor concerns; the framework is still actively used for agent benchmarking and leaderboard submissions.

Install

terminal-bench on PyPI

pip install terminal-bench

uv add terminal-bench

poetry add terminal-bench

Installing terminal-bench

Before you install

Low install friction with a pure-Python wheel distribution. The package is aging (322 days since release) but remains actively maintained; however, it requires Python 3.12+ and depends on 20 runtime packages including Docker, which adds system-level prerequisites beyond the Python ecosystem.

License in practice

License treatment is unclear—no SPDX identifier or raw license text is available in the metadata. Before relying on this package in a commercial or copyleft-sensitive context, verify the actual license from the project repository or documentation.

Quickstart

pip install terminal-bench

from terminal_bench import run

# Run benchmark with an agent and model
run(agent='terminus', model='anthropic/claude-3-7-latest', dataset_name='terminal-bench-core')

Requires Python 3.12+, Docker, and the uv package manager for full functionality. The execution harness also needs a sandboxed terminal environment to run tasks.

Verify before relying

• Whether the package is suitable for production use or remains experimental despite beta status
• Exact license terms and any restrictions on commercial or derivative use
• Performance characteristics and typical runtime for the ~100 tasks in the current dataset

Package facts

Evidence: terminal_bench-0.2.18-py3-none-any.whl

Tags

More Testing packages

Further reading