High partial scores hide near-total delivery failure in scientific AI agents

The gap between partial credit and actual delivery is the central finding here, and it is damning. Across twelve frontier models and three agent scaffolds, the best configurations completed just 20 of 97 released tasks—a Pass Rate of 20.6%—while average partial scores reached as high as 87.9 out of 100. That spread is not a measurement artifact. It is the point.

FrontierChallenge is a cross-domain benchmark of 300 end-to-end scientific workflows, of which 97 are publicly released. The six domains—quantum chemistry, molecular dynamics, materials characterization, analytical chemistry, life science, and electrochemistry/environment—each demand that an agent handle domain-specific software (ORCA, LAMMPS, AmberTools, and others), execute interdependent analytical steps, and deliver a coherent bundle of artifacts: code, tables, figures, and reports that must remain mutually consistent. A task is only a pass if the entire contract is satisfied. Partial progress, however impressive, does not count.

The electrochemistry and analytical chemistry results are the starkest illustration. Electrochemistry reached a maximum average score of 94.9—essentially full marks on partial credit—yet every single configuration achieved a Pass Rate of 0%. Analytical chemistry's best average score was 87.6, but only one configuration completed any task at all, at a 4% rate. Something is systematically breaking at the final delivery stage in these domains, and high intermediate scores offer no warning.

The failure analysis of 970 Claude Code trajectories makes the calibration problem concrete. Among non-passing runs, 75.5% ended with the model's final message claiming the work was complete. Completion language was actually more common in passing runs (90.1%), so it carries almost no diagnostic signal. Tool errors are similarly uninformative: they appeared in 80.7% of non-passing trajectories but also in 94.2% of passing ones. An agent that hits errors and recovers is not distinguishable from one that hits errors and quietly fails to deliver.

The benchmark's design choices matter here. Each task is packaged with fixed inputs, a defined execution environment, a stepwise scoring rubric, and a task-specific executable Grader. The 97 released tasks were randomly selected from the subset not requiring GPU evaluation; 203 tasks remain as a held-out set. The released set skews hard: 74 of the 97 tasks are classified Hard, 23 Medium. That composition is intentional—the benchmark is explicitly targeting the frontier, not measuring average capability.

What this work establishes is that contract-level delivery is a distinct capability, not a natural consequence of high partial scores. An agent that can navigate most of a complex scientific workflow and still miss the final cross-artifact consistency check is not close to reliable. The authors frame the implication directly: useful scientific agents will need explicit contract tracking and evidence-based completion checks, not just better reasoning about individual steps. The benchmark gives the field a concrete way to measure that gap.