AI IQ
Subscribe

IOI Cost Efficiency

Archived chart. IOI V1 archived; V2 uses a different harness and 2024–2026 tasks.

IOI Cost Efficiency
IOI Cost Efficiency
Archived chart. IOI V1 archived; V2 uses a different harness and 2024–2026 tasks.

How to read this chart

IOI publishes no per-run cost, so this chart falls back to each model's estimated effective cost per 1M I/O tokens (a runtime estimate from measured or imputed token usage, not a published task cost). Explicit reasoning/nonreasoning siblings are connected where both are public; every other model is a standalone point. Up and to the left is better.

Data sources
Artificial AnalysisARC PrizeVals Index v2 Cost per TestVals.aiVals.ai IOI
Archived chartIOI vs Effective CostArchived chart. IOI V1 archived; V2 uses a different harness and 2024–2026 tasks.Data: Artificial Analysis, ARC Prize, Vals Index v2 Cost per Test +2 moreOpen chartArchived chartIOI Benchmark ScoresArchived chart. IOI V1 archived; V2 uses a different harness and 2024–2026 tasks.Data: Vals.ai IOIOpen chartArchived chartProofBench Benchmark ScoresArchived chart. Historical results combine benchmark versions and are excluded from IQ. Use the separately versioned ProofBench 1.1 for current comparisons.Data: Vals.aiOpen chartArchived chartFrontierSWE ScoresArchived chart. V1 Dominance retired from Composite IQ and Software Engineering; V2 mean@5 requires a separate calibration.Data: FrontierSWEOpen chartMore / ExperimentalSWE-Bench Verified Benchmark ScoresHistorical data. Vals archived cohort; retired from default Software Engineering domain. Each model's SWE-Bench Verified score. Color = provider.Data: Vals.ai, LLM StatsOpen chartMore / ExperimentalTerminal-Bench 2.0 Benchmark ScoresHistorical data. Superseded by 2.1 and 4.0. Legacy Terminal-Bench 2.0 scores retained for historical comparison. Color = provider.Data: Terminal-BenchOpen chart