DeepSWE v1.1 Benchmark Scores
Each model's DeepSWE v1.1 Pass@1 score. Color = provider.
DeepSWE v1.1 Benchmark Scores
DeepSWE v1.1 Benchmark Scores
Each model's DeepSWE v1.1 Pass@1 score. Color = provider.
Controls:
How to read this chart
Bars rank models by the source-backed benchmark value used for this chart. Longer bars indicate higher published scores.
Data sources
DeepSWE v1.1 vs Effective Cost
DeepSWE v1.1 vs Effective Cost
X = effective cost (log). Y = DeepSWE v1.1 %. Color = provider.
Controls:
How to read this chart
Each point is a public model. The chart compares DeepSWE v1.1 % against Effective Cost (per 1M I/O Tokens), with color showing the model provider.
Data sources
IQ DimensionsBackend Engineering IQEach model's Backend Engineering IQ plotted on a standard normal IQ distributionData: Vals.ai, Cognition FrontierCode, Mercor APEX-SWE +7 moreOpen chartCoreAI Models on the IQ Bell CurveEach model's estimated IQ plotted on a standard normal IQ distributionData: ARC Prize, Epoch AI FrontierMath, Vals.ai +25 moreOpen chartIQ BenchmarksSWE-Bench Verified Benchmark ScoresEach model's SWE-Bench Verified score. Color = provider.Data: Vals.ai, LLM StatsOpen chartIQ BenchmarksLiveCodeBench Benchmark ScoresEach model's LiveCodeBench score. Color = provider.Data: Vals.aiOpen chartIQ BenchmarksSWE-Bench Pro Benchmark ScoresEach model's SWE-Bench Pro percentage score. Color = provider.Data: Scale Labs, Scale Labs SWE-Bench Pro Public DatasetOpen chartIQ BenchmarksAPEX-SWE ScoresMercor APEX-SWE overall Pass@1 scores for economically valuable software-engineering work. Color = provider.Data: Mercor APEX-SWEOpen chart