SWE-bench
Benchmark that tests whether language models and coding agents can resolve real GitHub issues, with multimodal and verified variants.
Updated
✨ Try AgentScout's free AI image generator →Good for: writing code, evaluating agents
Use cases
About SWE-bench
SWE-bench is listed in our evals & observability category and is aimed at builders. Benchmark that tests whether language models and coding agents can resolve real GitHub issues, with multimodal and verified variants. Pricing, as the maker states it: Free and open source (MIT). Maturity: Mature. We last checked the official link on 2026-10-21.
What is SWE-bench?
SWE-bench is the benchmark that tests whether language models and coding agents can resolve real GitHub issues: given a codebase and an issue, the system must produce a patch that makes the project's tests pass. It was introduced at ICLR 2024 and is one of the most cited coding-agent benchmarks.
Variants include SWE-bench Verified, a 500-task subset that engineers confirmed solvable (with OpenAI Preparedness), and SWE-bench Multimodal for visual software tasks, whose v2 with 480 tasks became fully open source in September 2026. Evaluation runs in containers locally, on Modal, or through the sb-cli cloud tool for leaderboard submissions.
Sources: the official README and our directory listing, checked 2026-10-21. Official site: swebench.com.
Who is SWE-bench for?
Researchers and teams who want to measure and compare coding models and agents on real-world issues.
SWE-bench at a glance
- Open source: Yes (MIT License)
- GitHub: SWE-bench/SWE-bench · 6,067 stars · 1,025 forks (as of Oct 21, 2026)
Key features
- Real GitHub issues with test-based grading
- SWE-bench Verified 500-task subset
- SWE-bench Multimodal for visual tasks
- Containerized evaluation harness
- Cloud runs via Modal and sb-cli
SWE-bench FAQ
What is SWE-bench?
SWE-bench is the benchmark that tests whether language models and coding agents can resolve real GitHub issues: given a codebase and an issue, the system must produce a patch that makes the project's tests pass. It was introduced at ICLR 2024 and is one of the most cited coding-agent benchmarks.
Who is SWE-bench for?
Researchers and teams who want to measure and compare coding models and agents on real-world issues.
Is SWE-bench free?
SWE-bench's pricing, as the maker states it: Free and open source (MIT). Check the official site for current plans.
Is SWE-bench open source?
Yes. SWE-bench is released under the MIT License, according to its GitHub repository.
What are alternatives to SWE-bench?
Similar tools include OSWorld, Tracely, agent-qa (Vostride), Openlayer MCP, Hamming AI and Latitude.
SWE-bench ratings and reviews
Be the first to rate SWE-bench: tap 1 to 5 stars below. A written review is optional.
SWE-bench alternatives
Other evals & observability tools in our directory. See all evals & observability tools →
OSWorld
Benchmark for multimodal computer-use agents on open-ended tasks in real desktop environments.
Tracely
Tracely is open-source, trace-native CI/CD for AI agents: it grades every production trace with evaluators, clusters failures into issues, freezes bad runs into replayable test cases and blocks pull requests that would reintroduce them.
agent-qa (Vostride)
A self-improving AI QA agent with persistent memory that tests web, Android and iOS apps from plain-English test steps, for developers and coding agents.
Openlayer MCP
Openlayer's MCP server, letting coding agents pull evaluation results, failing tests and monitoring data from Openlayer so they can debug and improve AI systems from the IDE.
Hamming AI
Testing and QA platform for voice and chat agents that generates scenarios and replays real calls.
Latitude
Observability platform for AI agents that captures traces, groups failures into signals and verifies fixes against real traces.
Popular searches with this tool
🏆 Ranked in: Best AI Agent Observability & Eval Tools
🧭 Task guides: AI eval and benchmark tools
Badge for makers
Built SWE-bench? Add this badge to your README or site. It links back to this page. About badges
