Benchmarks without benchmark soup.
Artificial Index stores each performance observation with the exact model, benchmark, benchmark version, harness/protocol, source, date and applicability. Raw percentages from different benchmarks are not averaged together. Cross-benchmark rollups require compatible within-benchmark peer percentiles and explicit current-model mapping.
Benchmark registry
GPQA
Graduate-level multiple-choice questions in biology, physics, and chemistry designed to be difficult even with web access.
Native metric: accuracy_pct · higher is better
Primary benchmark source ↗MMLU-Pro
A harder, reasoning-focused extension of MMLU with ten answer choices and reduced prompt sensitivity.
Native metric: accuracy_pct · higher is better
Primary benchmark source ↗AIME25
2025 American Invitational Mathematics Examination problems packaged for AI mathematical reasoning evaluation.
Native metric: accuracy_pct · higher is better
Primary benchmark source ↗LiveCodeBench
Continuously refreshed coding evaluation using recently released contest problems with code-generation, execution and repair scenarios.
Native metric: pass_at_1_pct · higher is better
Primary benchmark source ↗SWE-bench Verified
Human-validated subset of 500 real GitHub software-engineering tasks; comparable runs require the same agent harness.
Native metric: resolved_pct · higher is better
Primary benchmark source ↗MMMU-Pro
Robust multimodal reasoning benchmark derived from MMMU, including vision-only settings and harder answer choices.
Native metric: accuracy_pct · higher is better
Primary benchmark source ↗SimpleQA
Short, fact-seeking questions designed to measure factual correctness and hallucination behavior.
Native metric: correct_pct · higher is better
Primary benchmark source ↗BFCL V4
Berkeley Function Calling Leaderboard V4 evaluates tool/function calling, including multi-turn and agentic behavior.
Native metric: overall_accuracy_pct · higher is better
Primary benchmark source ↗GAIA
Real-world assistant tasks requiring reasoning, multimodality, browsing and tool use.
Native metric: accuracy_pct · higher is better
Primary benchmark source ↗OSWorld-Verified
Execution-based benchmark for multimodal agents performing real computer tasks across operating systems.
Native metric: success_pct · higher is better
Primary benchmark source ↗Initial independent snapshot
R14 starts with a small source-backed SWE-bench Verified Bash-Only snapshot to validate the ingestion, provenance and graph pipeline. These observations are model-level and do not yet roll into product headline scores.
| Model | Benchmark | Resolved | Avg. cost | Run date | Protocol |
|---|---|---|---|---|---|
| Claude 4.5 Opus (high reasoning) | swe-bench-verified | 76.8% | $0.75 | 2026-02-17 | mini-swe-agent-2.0.0-bash-only |
| Gemini 3 Flash (high reasoning) | swe-bench-verified | 75.8% | $0.36 | 2026-02-17 | mini-swe-agent-2.0.0-bash-only |
| MiniMax M2.5 (high reasoning) | swe-bench-verified | 75.8% | $0.07 | 2026-02-17 | mini-swe-agent-2.0.0-bash-only |
| Claude 4.6 Opus | swe-bench-verified | 75.6% | $0.55 | 2026-02-17 | mini-swe-agent-2.0.0-bash-only |
| GPT 5.2 Codex | swe-bench-verified | 72.8% | $0.45 | 2026-02-19 | mini-swe-agent-2.0.0-bash-only |
Artificial Index operational protocols
Artificial Index TTFT Protocol v1
Time to first output token measured server-side from request dispatch to first streamed model token.
Minimum samples: 30
Metric: ttft_ms · lower is better
Artificial Index Throughput Protocol v1
Generated output tokens divided by generation time after first token.
Minimum samples: 30
Metric: tokens_per_second · higher is better
Artificial Index Availability Protocol v1
Successful probe availability over a defined observation window.
Minimum samples: 168
Metric: uptime_pct · higher is better
Artificial Index Error Rate Protocol v1
Provider/server failure rate on a fixed, valid probe request set.
Minimum samples: 168
Metric: error_rate_pct · lower is better
Artificial Index Structured Output Protocol v1
Percentage of responses that validate against a fixed JSON Schema without repair.
Minimum samples: 100
Metric: schema_success_pct · higher is better
Artificial Index Tool-Use Probe v1
Controlled function-selection and argument-validity test set maintained by Artificial Index.
Minimum samples: 100
Metric: tool_success_pct · higher is better
Artificial Index Cost per Successful Task v1
Total benchmark execution cost divided by successful task count.
Minimum samples: 30
Metric: usd_per_success · lower is better
How scores become eligible
Exact observation
Record the exact model/version, benchmark, native score, source, date and protocol.
Comparable peer pool
At least five observations must share the benchmark/version/protocol context before a percentile is calculated.
Current model mapping
A researcher must explicitly verify that the observation applies to the AI product/model being scored now.
Domain rollup
Intelligence requires at least three independent compatible domains. Speed and reliability use their own operational protocols.
R17 independent measurement pilot
R17 adds a 9-model independent comparison cohort. Seven Artificial Index system records now have a verified representative-model mapping and enough independent intelligence, throughput and TTFT data to produce a Measured Evidence score. Intelligence carries 70% of the score; throughput and latency carry 15% each.