Terminal-Bench-Science measures what agents can do, not what models know

Most AI benchmarks test what a model knows. Terminal-Bench-Science tests what an agent can do — specifically, whether it can execute the kind of multi-step, tool-heavy workflows that working scientists actually run. The 70 tasks currently in the benchmark span life sciences, physical sciences, earth sciences, mathematics, and engineering, each authored by domain experts and verified to pass an oracle solution before any agent sees them.

The design philosophy here is worth examining closely. Tasks must produce outcomes that are objectively verifiable in a terminal environment. That constraint rules out a large class of plausible-sounding but unverifiable science tasks — summarizing a paper, proposing a hypothesis, explaining a method. What remains is executable: run this analysis, produce this output file, reproduce this numerical result. The benchmark is honest about what it can measure, and that honesty is more useful than breadth.

The quality control pipeline is unusually rigorous for an open community effort. Every proposed task moves through a formal proposal form, then a pull request, then automated checks covering path validation, Dockerfile sanity, a 39-criteria implementation rubric, TF-IDF duplicate detection, Docker build verification, oracle and no-op validation, multi-agent trial runs, and adversarial reward-hack detection. Human domain reviewers and a bar-raiser approval gate follow. That is a longer gauntlet than many published benchmarks impose on their own authors.

The benchmark runs on top of Harbor, which handles sandboxed execution across Modal or Daytona environments. The quickstart is genuinely minimal: install Harbor, point it at the dataset, pass an agent and model flag. Running oracle solutions five times to confirm environmental stability before evaluating an agent is a sensible sanity check that most benchmark documentation omits.

The community structure is also notable. Contributors span Stanford, MIT, Oxford, ETH Zurich, Caltech, and a long list of other institutions. Scientific advisors include researchers working on ecology, astrophysics, neuroscience, and computational biology. The breadth of that contributor list is what makes the domain coverage credible — a benchmark about scientific workflows written entirely by ML researchers would look very different.

The version 0.2 deadline of October 5, 2026 signals that this is a living artifact with a defined release cadence, not a one-shot publication. The concept DOI always resolves to the latest release, which is the right call for a benchmark that explicitly intends to evolve alongside frontier models.

The 70-task count is modest. The benchmark acknowledges it is growing toward 100 or more tasks, which means coverage gaps are real and current leaderboard positions should be read with that in mind. Whether the terminal-only constraint captures enough of scientific research practice to generalize is a fair question — plenty of consequential scientific work happens in notebooks, GUIs, and collaborative documents that a terminal agent never touches. But for the slice of scientific computing that is scriptable and reproducible, this is a serious attempt to measure something real.