AI IQ
Consultation Subscribe

ProgramBench

A bird's-eye view of the benchmark: what it measures and every AI IQ chart built on it.

About ProgramBench

ProgramBench measures robust program synthesis; Almost Resolved is the share of tasks passing at least 95% of hidden tests. The efficiency view compares each published configuration with effective cost.

ProgramBench Cost Efficiency
ProgramBench Cost Efficiency
ProgramBench Almost Resolved versus runtime effective cost. Explicit reasoning and nonreasoning variants are connected; other published configurations remain standalone points. Color = provider.

How to read this chart

This chart compares source-backed ProgramBench configurations with runtime effective cost. Multiple reasoning levels for the same model are connected; models with one available level remain standalone points. Up and to the left is better.

Data sources
Artificial AnalysisARC PrizeVals.aiVals.ai ProgramBench
ProgramBench Scores
ProgramBench Scores
Almost Resolved score: share of ProgramBench tasks passing at least 95% of hidden tests. Color = provider.

How to read this chart

Bars rank models by the source-backed benchmark value used for this chart. Longer bars indicate higher published scores.

Data sources
Vals.ai ProgramBench
ProgramBench vs Effective Cost
ProgramBench vs Effective Cost
X = effective cost (log). Y = ProgramBench Almost Resolved. Color = provider.
Controls:
ProgramBench1:1Cost

How to read this chart

Each point is a public model. The chart compares Almost Resolved against Effective Cost (per 1M I/O Tokens), with color showing the model provider.

Data sources
Artificial AnalysisARC PrizeVals.aiVals.ai ProgramBench

← All benchmarks