---
id: swebench
version: "4.1.0"
license: MIT License Copyright (c) 2023 Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, Karthik R Narasimhan Permission is hereby granted, free of charge, to any person… (full text in the JSON record)
license_treatment: permissive
maintenance: active
---

# swebench — The official SWE-bench package - a benchmark for evaluating LMs on software engineering

License: permissive · Maintenance: active · Popularity: top 1,000 on PyPI

## Install

pip install swebench
uv add swebench
poetry add swebench

## Description

<p align="center">
  <a href="http://swe-bench.github.io">
    <img src="docs/assets/figures/swellama_banner.svg" style="height: 10em" alt="Kawi the SWE-Llama" />
  </a>
</p>

<p align="center"><strong>[&nbsp;<a href="https://swebench.com/SWE-bench/">Read the Docs</a>&nbsp;]</strong></p>

<p align="center">
  <a href="docs/other_languages/README_JP.md">日本語</a> |
  <a href="docs/other_languages/README_CN.md">中文简体</a> |
  <a href="docs/other_languages/README_TW.md">中文繁體</a>
</p>

<p align="center">
    <a href="https://www.python.org/">
        <img alt="Build" src="https://img.shields.io/badge/Python-3.8+-1f425f.svg?color=purple">
    </a>
    <a href="https://copyright.princeton.edu/policy">
        <img alt="License" src="https://img.shields.io/badge/License-MIT-blue">
    </a>
    <a href="https://badge.fury.io/py/swebench">
        <img src="https://badge.fury.io/py/swebench.svg">
    </a>
</p>

---

Code and data for the following works:
* [ICLR 2025] <a href="https://arxiv.org/abs/2410.03859">SWE-bench Multimodal: Do AI Systems Generalize to Visual Software Domains?</a>
* [ICLR 2024 Oral] <a href="https://arxiv.org/abs/2310.06770">SWE-bench: Can Language Models Resolve...

## AI interpretation — verify before relying

SWE-bench is a benchmark for evaluating language models on real-world GitHub software issues, where models generate patches to resolve described problems in codebases.

Verdict: Actively maintained, well-resourced benchmark with no known vulnerabilities and permissive licensing. Suitable for evaluating LM code-generation capabilities, but evaluation is resource-intensive and requires Docker infrastructure setup.

[View on SkillFed](https://skillfed.io/packages/swebench) · [View on PyPI](https://pypi.org/project/swebench/)