SWE-bench Science: Can Coding Agents Resolve Engineering Tasks in Science?

Scientific software is not just code that happens to run in a lab — it is part of the instrument itself. A bug in a climate model or a genomics pipeline does not merely crash a program; it corrupts the evidence. SWE-bench Science takes that premise seriously and builds a benchmark around it: 119 tasks drawn from 98 GitHub repositories spanning 20 scientific domains, organized into three categories — issue-driven repair, expert-exploratory investigation, and engineering integration.

The headline result is blunt. The best agent tested, Claude Code with Opus-5 at maximum settings, clears a pass@1 below 50%. That is not a rounding error or a quirk of a single domain. It is a ceiling that holds across the benchmark, and it matters because the tasks are not contrived puzzles — they come from real repositories where correctness has downstream consequences for scientific conclusions.

The paper's more useful contribution is its taxonomy of failure. Four mechanisms recur: agents lack the domain knowledge to understand what a correct repair even looks like; they explore superficially and patch symptoms rather than causes; they fix one site while missing related breakage elsewhere in the system; and they fail to generalize a scientific principle beyond the specific case they observed. These are not random errors. They are structured, and that structure suggests where agent design needs to improve.

The ablation study is the sharpest piece of analysis. The authors stripped explicit scientific guidance from tasks while keeping the repository and executable context intact, then compared performance. The finding cuts against a simple story: scientific context is not uniformly helpful. When the guidance is well-grounded, it focuses the agent, improves average performance, and reduces token consumption. When it is poorly aligned with the actual problem, it anchors the agent to a wrong frame and can make exact repair success worse. That asymmetry has real implications for how retrieval-augmented or context-injected agent systems should be designed — more context is not always better, and the quality of domain framing matters as much as its presence.

The benchmark covers a wide spread of scientific fields, which makes it harder to dismiss as domain-specific. It also makes it harder to game with narrow fine-tuning. Whether the 119-task scale is sufficient to draw statistically stable conclusions about individual failure modes is a fair question the abstract does not fully address, but as a diagnostic instrument for understanding where agents break down on consequential code, it is more carefully constructed than most.