swebench

The official SWE-bench package - a benchmark for evaluating LMs on software engineering

swebench Permissive license MIT License Copyright (c) 2023 Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, Karthik R Narasimhan Permission is hereby granted, free of charge, to any person… (full text in the JSON record) Active ★ 5,632 v4.1.0 released 2025-09-11

Install

swebench on PyPI

pip

pip install swebench

uv

uv add swebench

poetry

poetry add swebench

Package facts

License	MIT License Copyright (c) 2023 Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, Karthik R Narasimhan Permission is hereby granted, free of charge, to any person… (full text in the JSON record) (permissive)
Python support	supports the current Python release (>=3.10)
Install friction	low — pure-Python wheel
Runtime dependencies	14 — beautifulsoup4, chardet, datasets, docker, ghapi, GitPython, modal, pre-commit, python-dotenv, requests, rich, tenacity, tqdm, unidiff
Maintenance	actively maintained — 336 days since the last release
Last repo commit	2026-08-13
First released	2023-11-01
Popularity	one of the top 1,000 most-downloaded packages on PyPI (30-day window, as of 2026-08-13)
Known vulnerabilities	none known (OSV.dev, checked 2026-08-13)

Evidence: swebench-4.1.0-py3-none-any.whl

Keywords: nlp, benchmark, code

License :: OSI Approved :: MIT LicenseOperating System :: OS IndependentProgramming Language :: Python :: 3 :: OnlyProgramming Language :: Python :: 3.10Programming Language :: Python :: 3.11

About swebench

from the package's own PyPI description — quoted content, verbatim

[ <a href="https://swebench.com/SWE-bench/">Read the Docs</a> ]

Code and data for the following works: * [ICLR 2025] <a href="https://arxiv.org/abs/2410.03859">SWE-bench Multimodal: Do AI Systems Generalize to Visual Software Domains?</a> * [ICLR 2024 Oral] <a href="https://arxiv.org/abs/2310.06770">SWE-bench: Can Language Models Resolve...

Read as markdown · JSON record · Source repository · Docs

AI interpretation — verify before relying

AI-generated interpretation of the package facts above; every digit, version, license, or vulnerability id it cites is grounded in the facts already shown on this page

SWE-bench is a benchmark for evaluating language models on real-world GitHub software issues, where models generate patches to resolve described problems in codebases.

Low friction installation with a pure-Python wheel. Active maintenance (latest release 2025-09-11) and strong community signal (5632 stars). Requires Docker for reproducible evaluations and substantial resources (120GB storage, 16GB RAM, 8 CPU cores recommended).

MIT License permits unrestricted use, modification, and distribution with minimal restrictions, making it suitable for academic and commercial applications.

Usage

pip install swebench
from datasets import load_dataset
swebench = load_dataset('princeton-nlp/SWE-bench', split='test')
python -m swebench.harness.run_evaluation --dataset_name princeton-nlp/SWE-bench_Lite --predictions_path <path> --max_workers 8 --run_id my_eval

Docker installation required; x86_64 machine with ≥120GB free storage, 16GB RAM, 8 CPU cores strongly recommended; Python ≥3.10 required.

Verdict: Actively maintained, well-resourced benchmark with no known vulnerabilities and permissive licensing. Suitable for evaluating LM code-generation capabilities, but evaluation is resource-intensive and requires Docker infrastructure setup.

Needs verification

Whether the 14 runtime dependencies (docker, modal, ghapi, etc.) introduce any transitive security concerns beyond the OSV scan
Specific performance characteristics and typical evaluation runtime on recommended hardware
Whether ARM-based evaluation (experimental support noted) is production-ready for M-series Macs

llm code generation benchmarkgithub issue resolution evaluationsoftware engineering benchmarklanguage model patch generationcode bug fixing datasetai system code repairml model software testing