Benchmark structure
Understand the study.
Inspect the evidence.
Move between the benchmark methodology, evaluated configurations and exercise-level research records.
21 configurations60 exercises1,260 matched cells2 subjects
01
Technical comparison
Evaluated configurations
Technical specifications follow the benchmark record. Each API rate carries its own verification date; benchmark rates refer to the frozen snapshot of 15 Oct 2026.
| Identity | Architecture | Limits | Benchmark setup | Pricing · USD per 1M tokens | |||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| Model / API identifier | Public release | Parameters | Open weights | Architecture / model class | Context window | Maximum output | Knowledge / training cutoff | Reasoning capability / benchmark mode | Official links | Latest verified API rate | Benchmark rate |
GPT-6.1 SolOpenAIgpt-6.1-sol | 29 Sep 2026 · GA | Not disclosed | No | Closed Sol-tier reasoning model; architecture undisclosed | 1,050,000 | 128,000 | 30 Apr 2026 knowledge | Configurable effort: low to max; benchmark: xhigh | Latest verified API rateInput $2 · Output $10Cached input $0.10Standard tier below the long-context threshold.Verified 15 Oct 2026 | BenchmarkInput $2 · Output $10Snapshot 15 Oct 2026 | |
GPT-6 LunaOpenAIgpt-6-luna | 22 Sep 2026 · GA | Not disclosed | No | Closed efficient reasoning model; architecture undisclosed | 1,050,000 | 128,000 | 18 May 2026 knowledge | Configurable effort: none to max; benchmark: xhigh | Latest verified API rateInput $0.10 · Output $0.50Cached input $0.01Standard tier below the long-context threshold.Verified 15 Oct 2026 | BenchmarkInput $0.10 · Output $0.50Snapshot 15 Oct 2026 | |
GPT-5.6 SolOpenAIgpt-5.6-sol | 9 Jul 2026 · GA; preview 26 Jun 2026 | Not disclosed | No | Closed frontier flagship; architecture undisclosed | 1,050,000 | 128,000 | 16 Feb 2026 knowledge | Configurable effort: none to max; benchmark: xhigh | Latest verified API rateInput $4 · Output $20Cached input $0.40Promotional rate; long-context requests use higher rates.Verified 15 Oct 2026 | BenchmarkInput $5 · Output $30Snapshot 15 Oct 2026 | |
OpenAI o3OpenAIo3-2025-04-16 | 16 Apr 2025 · dated snapshot | Not disclosed | No | Closed o-series reasoning model; architecture undisclosed | 200,000 | 100,000 | 1 Jun 2024 knowledge | Native inference-time reasoning; benchmark: high | BenchmarkInput $2 · Output $8Snapshot 15 Oct 2026 | ||
OpenAI GPT-4.1OpenAIgpt-4.1-2025-04-14 | 14 Apr 2025 · dated snapshot | Not disclosed | No | Closed general-purpose non-reasoning model | 1,047,576 | 32,768 | 1 Jun 2024 knowledge | No native reasoning step; benchmark: none | BenchmarkInput $2 · Output $8Snapshot 15 Oct 2026 | ||
Claude Fable 5Anthropicclaude-fable-5 | 9 Jun 2026 · GA; restored 1 Jul 2026 | Not disclosed | No | Closed Mythos-class safeguarded model | 1,000,000 | 128,000 | Jan 2026 reliable/training | Always-on adaptive thinking; benchmark: effort high | Latest verified API rateInput $10 · Output $50Verified 15 Oct 2026 | BenchmarkInput $10 · Output $50Snapshot 15 Oct 2026 | |
Claude Opus 5.5Anthropicclaude-opus-5-5 | 22 Sep 2026 · GA | Not disclosed | No | Closed Opus-tier hybrid reasoning model | 1,000,000 | 128,000 | Jun 2026 reliable/training | Always-on adaptive thinking; benchmark: effort high | BenchmarkInput $4 · Output $20Snapshot 15 Oct 2026 | ||
Claude Sonnet 5.5Anthropicclaude-sonnet-5-5 | 28 Sep 2026 · GA | Not disclosed | No | Closed Sonnet-tier hybrid reasoning model | 1,000,000 | 128,000 | Jun 2026 reliable/training | Adaptive thinking; benchmark: effort high | BenchmarkInput $2 · Output $10Snapshot 15 Oct 2026 | ||
Claude Sonnet 4.5Anthropicclaude-sonnet-4-5-20250929 | 29 Sep 2025 · dated snapshot | Not disclosed | No | Closed Sonnet-tier hybrid reasoning model | 200,000 | 64,000 | Not disclosed | Optional manual extended thinking; benchmark: provider default | Latest verified API rateInput $3 · Output $15Verified 15 Oct 2026 | BenchmarkInput $3 · Output $15Snapshot 15 Oct 2026 | |
Claude Haiku 4.5Anthropicclaude-haiku-4-5-20251001 | 15 Oct 2025 · API snapshot 1 Oct 2025 | Not disclosed | No | Closed small and low-latency tier | 200,000 | 64,000 | Feb/Jul 2025 reliable/training | Manual extended thinking, off by default; benchmark: not requested | Latest verified API rateInput $1 · Output $5Verified 15 Oct 2026 | BenchmarkInput $1 · Output $5Snapshot 15 Oct 2026 | |
Gemini 3.5 FlashGooglegemini-3.5-flash | 19 May 2026 · GA | Not disclosed | No | Closed multimodal Flash-tier model | 1,048,576 | 65,536 | Jan 2025 knowledge | Configurable native thinking; benchmark: high | BenchmarkInput $1.50 · Output $9Snapshot 15 Oct 2026 | ||
Gemini 3.1 Pro PreviewGooglegemini-3.1-pro-preview | 19 Feb 2026 · public preview | Not disclosed | No | Closed multimodal Pro-tier model | 1,048,576 | 65,536 | Jan 2025 knowledge | Configurable native thinking; benchmark: high | BenchmarkInput $2 · Output $12Benchmark requests used the standard context tier.Snapshot 15 Oct 2026 | ||
Gemma 4 31B ITGooglegemma-4-31b-it | 31 Mar 2026 · API 2 Apr 2026 | 30.7B total; dense; 60 layers; about 550M vision encoder | Yes | Dense decoder-only transformer; hybrid local/global attention | 256,000 | Not specified | Jan 2025 training | Configurable thinking tokens; benchmark: disabled | BenchmarkInput $0 · Output $0Configured campaign rate; not a universal market price.Snapshot 15 Oct 2026 | ||
DeepSeek V4.1 FlashDeepSeekdeepseek-flash | 10 Sep 2026 · API observed 16 Sep 2026 | 552B backbone + 196B conditional memory; 8B / 16B active in prefill / decode | Yes | Multimodal Mixture-of-Experts; causal encoder-decoder; Compressed Sparse Attention 2 | 1,000,000 | 384,000 | Not disclosed | Dual thinking/non-thinking model; benchmark: thinking high | Latest verified API rateOff-peak: $0.15 · $0.60Peak: $0.30 · $1.20Input cache-miss · output; time-dependent pricing.Verified 16 Sep 2026 | BenchmarkInput $0.30 · Output $1.20Conservative peak-rate estimate with uncached input; not an actual invoice.Snapshot 15 Oct 2026 | |
DeepSeek V4 FlashDeepSeekdeepseek-v4-flash - thinking | 24 Apr 2026 · V4 preview | 284B total; 13B active per token | Yes | Mixture-of-Experts; DeepSeek Sparse Attention | 1,000,000 | 384,000 | Not disclosed | Dual thinking/non-thinking model; benchmark: thinking high | Latest verified API rateOff-peak: $0.22 · $0.66Peak: $0.44 · $1.32Input cache-miss · output; time-dependent pricing.Verified 15 Oct 2026 | BenchmarkInput $0.14 · Output $0.28Snapshot 15 Oct 2026 | |
DeepSeek V4 Flash (no thinking)DeepSeekdeepseek-v4-flash - no thinking | 24 Apr 2026 · same weights as above | 284B total; 13B active per token | Yes | Mixture-of-Experts; DeepSeek Sparse Attention | 1,000,000 | 384,000 | Not disclosed | Dual thinking/non-thinking model; benchmark: disabled | Latest verified API rateOff-peak: $0.22 · $0.66Peak: $0.44 · $1.32Input cache-miss · output; time-dependent pricing.Verified 15 Oct 2026 | BenchmarkInput $0.14 · Output $0.28Snapshot 15 Oct 2026 | |
DeepSeek V4 ProDeepSeekdeepseek-v4-pro | 24 Apr 2026 · V4 preview | 1.6T total; 49B active per token | Yes | Mixture-of-Experts; DeepSeek Sparse Attention | 1,000,000 | 384,000 | Not disclosed | Dual thinking/non-thinking model; benchmark: thinking high | Latest verified API rateOff-peak: $0.66 · $1.98Peak: $1.32 · $3.96Input cache-miss · output; time-dependent pricing.Verified 15 Oct 2026 | BenchmarkInput $0.435 · Output $0.87Snapshot 15 Oct 2026 | |
Grok 4.5xAIgrok-4.5 | 16 Jul 2026 · public launch | Not disclosed | No | Closed frontier model; architecture undisclosed | 500,000 | Not specified | 1 Feb 2026 knowledge | Configurable low/medium/high; benchmark: high | BenchmarkInput $2 · Output $6Snapshot 15 Oct 2026 | ||
Grok 4.3xAIgrok-4.3 | By 6 May 2026* · first dated official use | Not disclosed | No | Closed general-purpose model; architecture undisclosed | 1,000,000 | Not specified | Not disclosed | Configurable none/low/medium/high; benchmark: high | Latest verified API rateInput $1.25 · Output $2.50Cached $0.20 · >200k: $2.50 / $5Verified 15 Oct 2026 | BenchmarkInput $1.25 · Output $2.50Snapshot 15 Oct 2026 | |
Mistral Medium 3.5MistralRequested alias mistral-medium-latestResolved model mistral-medium-3-5 | API snapshot observed 13 Sep 2026 | Not disclosed | No | Closed Medium-tier model; architecture undisclosed | 256,000 | Not specified | Not disclosed | Configurable reasoning effort; benchmark: high | Latest verified API rateInput $1.50 · Output $7.50Verified 14 Sep 2026 | BenchmarkInput $1.50 · Output $7.50Campaign rate recorded with the Mistral benchmark runs.Snapshot 15 Oct 2026 | |
Mistral Small 4MistralRequested alias mistral-small-latestResolved model mistral-small-2603 | API snapshot observed 13 Sep 2026 | Not disclosed | No | Closed Small-tier model; architecture undisclosed | 256,000 | Not specified | Not disclosed | Configurable reasoning effort; benchmark: high | Latest verified API rateInput $0.15 · Output $0.60Verified 14 Sep 2026 | BenchmarkInput $0.15 · Output $0.60Campaign rate recorded with the Mistral benchmark runs.Snapshot 15 Oct 2026 | |
GPT-6.1 SolOpenAIgpt-6.1-sol+
Latest verified API rateInput $2 · Output $10Cached input $0.10Standard tier below the long-context threshold.Verified 15 Oct 2026
BenchmarkInput $2 · Output $10Snapshot 15 Oct 2026
- Public release
- 29 Sep 2026 · GA
- Parameters
- Not disclosed
- Open weights
- No
- Architecture / model class
- Closed Sol-tier reasoning model; architecture undisclosed
- Reasoning capability / benchmark mode
- Configurable effort: low to max; benchmark: xhigh
- Context window
- 1,050,000
- Maximum output
- 128,000
- Knowledge / training cutoff
- 30 Apr 2026 knowledge
GPT-6 LunaOpenAIgpt-6-luna+
Latest verified API rateInput $0.10 · Output $0.50Cached input $0.01Standard tier below the long-context threshold.Verified 15 Oct 2026
BenchmarkInput $0.10 · Output $0.50Snapshot 15 Oct 2026
- Public release
- 22 Sep 2026 · GA
- Parameters
- Not disclosed
- Open weights
- No
- Architecture / model class
- Closed efficient reasoning model; architecture undisclosed
- Reasoning capability / benchmark mode
- Configurable effort: none to max; benchmark: xhigh
- Context window
- 1,050,000
- Maximum output
- 128,000
- Knowledge / training cutoff
- 18 May 2026 knowledge
GPT-5.6 SolOpenAIgpt-5.6-sol+
Latest verified API rateInput $4 · Output $20Cached input $0.40Promotional rate; long-context requests use higher rates.Verified 15 Oct 2026
BenchmarkInput $5 · Output $30Snapshot 15 Oct 2026
- Public release
- 9 Jul 2026 · GA; preview 26 Jun 2026
- Parameters
- Not disclosed
- Open weights
- No
- Architecture / model class
- Closed frontier flagship; architecture undisclosed
- Reasoning capability / benchmark mode
- Configurable effort: none to max; benchmark: xhigh
- Context window
- 1,050,000
- Maximum output
- 128,000
- Knowledge / training cutoff
- 16 Feb 2026 knowledge
OpenAI o3OpenAIo3-2025-04-16+
BenchmarkInput $2 · Output $8Snapshot 15 Oct 2026
- Public release
- 16 Apr 2025 · dated snapshot
- Parameters
- Not disclosed
- Open weights
- No
- Architecture / model class
- Closed o-series reasoning model; architecture undisclosed
- Reasoning capability / benchmark mode
- Native inference-time reasoning; benchmark: high
- Context window
- 200,000
- Maximum output
- 100,000
- Knowledge / training cutoff
- 1 Jun 2024 knowledge
OpenAI GPT-4.1OpenAIgpt-4.1-2025-04-14+
BenchmarkInput $2 · Output $8Snapshot 15 Oct 2026
- Public release
- 14 Apr 2025 · dated snapshot
- Parameters
- Not disclosed
- Open weights
- No
- Architecture / model class
- Closed general-purpose non-reasoning model
- Reasoning capability / benchmark mode
- No native reasoning step; benchmark: none
- Context window
- 1,047,576
- Maximum output
- 32,768
- Knowledge / training cutoff
- 1 Jun 2024 knowledge
Claude Fable 5Anthropicclaude-fable-5+
Latest verified API rateInput $10 · Output $50Verified 15 Oct 2026
BenchmarkInput $10 · Output $50Snapshot 15 Oct 2026
- Public release
- 9 Jun 2026 · GA; restored 1 Jul 2026
- Parameters
- Not disclosed
- Open weights
- No
- Architecture / model class
- Closed Mythos-class safeguarded model
- Reasoning capability / benchmark mode
- Always-on adaptive thinking; benchmark: effort high
- Context window
- 1,000,000
- Maximum output
- 128,000
- Knowledge / training cutoff
- Jan 2026 reliable/training
Claude Opus 5.5Anthropicclaude-opus-5-5+
BenchmarkInput $4 · Output $20Snapshot 15 Oct 2026
- Public release
- 22 Sep 2026 · GA
- Parameters
- Not disclosed
- Open weights
- No
- Architecture / model class
- Closed Opus-tier hybrid reasoning model
- Reasoning capability / benchmark mode
- Always-on adaptive thinking; benchmark: effort high
- Context window
- 1,000,000
- Maximum output
- 128,000
- Knowledge / training cutoff
- Jun 2026 reliable/training
Claude Sonnet 5.5Anthropicclaude-sonnet-5-5+
BenchmarkInput $2 · Output $10Snapshot 15 Oct 2026
- Public release
- 28 Sep 2026 · GA
- Parameters
- Not disclosed
- Open weights
- No
- Architecture / model class
- Closed Sonnet-tier hybrid reasoning model
- Reasoning capability / benchmark mode
- Adaptive thinking; benchmark: effort high
- Context window
- 1,000,000
- Maximum output
- 128,000
- Knowledge / training cutoff
- Jun 2026 reliable/training
Claude Sonnet 4.5Anthropicclaude-sonnet-4-5-20250929+
Latest verified API rateInput $3 · Output $15Verified 15 Oct 2026
BenchmarkInput $3 · Output $15Snapshot 15 Oct 2026
- Public release
- 29 Sep 2025 · dated snapshot
- Parameters
- Not disclosed
- Open weights
- No
- Architecture / model class
- Closed Sonnet-tier hybrid reasoning model
- Reasoning capability / benchmark mode
- Optional manual extended thinking; benchmark: provider default
- Context window
- 200,000
- Maximum output
- 64,000
- Knowledge / training cutoff
- Not disclosed
Claude Haiku 4.5Anthropicclaude-haiku-4-5-20251001+
Latest verified API rateInput $1 · Output $5Verified 15 Oct 2026
BenchmarkInput $1 · Output $5Snapshot 15 Oct 2026
- Public release
- 15 Oct 2025 · API snapshot 1 Oct 2025
- Parameters
- Not disclosed
- Open weights
- No
- Architecture / model class
- Closed small and low-latency tier
- Reasoning capability / benchmark mode
- Manual extended thinking, off by default; benchmark: not requested
- Context window
- 200,000
- Maximum output
- 64,000
- Knowledge / training cutoff
- Feb/Jul 2025 reliable/training
Gemini 3.5 FlashGooglegemini-3.5-flash+
BenchmarkInput $1.50 · Output $9Snapshot 15 Oct 2026
- Public release
- 19 May 2026 · GA
- Parameters
- Not disclosed
- Open weights
- No
- Architecture / model class
- Closed multimodal Flash-tier model
- Reasoning capability / benchmark mode
- Configurable native thinking; benchmark: high
- Context window
- 1,048,576
- Maximum output
- 65,536
- Knowledge / training cutoff
- Jan 2025 knowledge
Gemini 3.1 Pro PreviewGooglegemini-3.1-pro-preview+
BenchmarkInput $2 · Output $12Benchmark requests used the standard context tier.Snapshot 15 Oct 2026
- Public release
- 19 Feb 2026 · public preview
- Parameters
- Not disclosed
- Open weights
- No
- Architecture / model class
- Closed multimodal Pro-tier model
- Reasoning capability / benchmark mode
- Configurable native thinking; benchmark: high
- Context window
- 1,048,576
- Maximum output
- 65,536
- Knowledge / training cutoff
- Jan 2025 knowledge
Gemma 4 31B ITGooglegemma-4-31b-it+
BenchmarkInput $0 · Output $0Configured campaign rate; not a universal market price.Snapshot 15 Oct 2026
- Public release
- 31 Mar 2026 · API 2 Apr 2026
- Parameters
- 30.7B total; dense; 60 layers; about 550M vision encoder
- Open weights
- Yes
- Architecture / model class
- Dense decoder-only transformer; hybrid local/global attention
- Reasoning capability / benchmark mode
- Configurable thinking tokens; benchmark: disabled
- Context window
- 256,000
- Maximum output
- Not specified
- Knowledge / training cutoff
- Jan 2025 training
DeepSeek V4.1 FlashDeepSeekdeepseek-flash+
Latest verified API rateOff-peak: $0.15 · $0.60Peak: $0.30 · $1.20Input cache-miss · output; time-dependent pricing.Verified 16 Sep 2026
BenchmarkInput $0.30 · Output $1.20Conservative peak-rate estimate with uncached input; not an actual invoice.Snapshot 15 Oct 2026
- Public release
- 10 Sep 2026 · API observed 16 Sep 2026
- Parameters
- 552B backbone + 196B conditional memory; 8B / 16B active in prefill / decode
- Open weights
- Yes
- Architecture / model class
- Multimodal Mixture-of-Experts; causal encoder-decoder; Compressed Sparse Attention 2
- Reasoning capability / benchmark mode
- Dual thinking/non-thinking model; benchmark: thinking high
- Context window
- 1,000,000
- Maximum output
- 384,000
- Knowledge / training cutoff
- Not disclosed
DeepSeek V4 FlashDeepSeekdeepseek-v4-flash - thinking+
Latest verified API rateOff-peak: $0.22 · $0.66Peak: $0.44 · $1.32Input cache-miss · output; time-dependent pricing.Verified 15 Oct 2026
BenchmarkInput $0.14 · Output $0.28Snapshot 15 Oct 2026
- Public release
- 24 Apr 2026 · V4 preview
- Parameters
- 284B total; 13B active per token
- Open weights
- Yes
- Architecture / model class
- Mixture-of-Experts; DeepSeek Sparse Attention
- Reasoning capability / benchmark mode
- Dual thinking/non-thinking model; benchmark: thinking high
- Context window
- 1,000,000
- Maximum output
- 384,000
- Knowledge / training cutoff
- Not disclosed
DeepSeek V4 Flash (no thinking)DeepSeekdeepseek-v4-flash - no thinking+
Latest verified API rateOff-peak: $0.22 · $0.66Peak: $0.44 · $1.32Input cache-miss · output; time-dependent pricing.Verified 15 Oct 2026
BenchmarkInput $0.14 · Output $0.28Snapshot 15 Oct 2026
- Public release
- 24 Apr 2026 · same weights as above
- Parameters
- 284B total; 13B active per token
- Open weights
- Yes
- Architecture / model class
- Mixture-of-Experts; DeepSeek Sparse Attention
- Reasoning capability / benchmark mode
- Dual thinking/non-thinking model; benchmark: disabled
- Context window
- 1,000,000
- Maximum output
- 384,000
- Knowledge / training cutoff
- Not disclosed
DeepSeek V4 ProDeepSeekdeepseek-v4-pro+
Latest verified API rateOff-peak: $0.66 · $1.98Peak: $1.32 · $3.96Input cache-miss · output; time-dependent pricing.Verified 15 Oct 2026
BenchmarkInput $0.435 · Output $0.87Snapshot 15 Oct 2026
- Public release
- 24 Apr 2026 · V4 preview
- Parameters
- 1.6T total; 49B active per token
- Open weights
- Yes
- Architecture / model class
- Mixture-of-Experts; DeepSeek Sparse Attention
- Reasoning capability / benchmark mode
- Dual thinking/non-thinking model; benchmark: thinking high
- Context window
- 1,000,000
- Maximum output
- 384,000
- Knowledge / training cutoff
- Not disclosed
Grok 4.5xAIgrok-4.5+
BenchmarkInput $2 · Output $6Snapshot 15 Oct 2026
- Public release
- 16 Jul 2026 · public launch
- Parameters
- Not disclosed
- Open weights
- No
- Architecture / model class
- Closed frontier model; architecture undisclosed
- Reasoning capability / benchmark mode
- Configurable low/medium/high; benchmark: high
- Context window
- 500,000
- Maximum output
- Not specified
- Knowledge / training cutoff
- 1 Feb 2026 knowledge
Grok 4.3xAIgrok-4.3+
Latest verified API rateInput $1.25 · Output $2.50Cached $0.20 · >200k: $2.50 / $5Verified 15 Oct 2026
BenchmarkInput $1.25 · Output $2.50Snapshot 15 Oct 2026
- Public release
- By 6 May 2026* · first dated official use
- Parameters
- Not disclosed
- Open weights
- No
- Architecture / model class
- Closed general-purpose model; architecture undisclosed
- Reasoning capability / benchmark mode
- Configurable none/low/medium/high; benchmark: high
- Context window
- 1,000,000
- Maximum output
- Not specified
- Knowledge / training cutoff
- Not disclosed
Mistral Medium 3.5MistralRequested alias mistral-medium-latestResolved model mistral-medium-3-5+
Latest verified API rateInput $1.50 · Output $7.50Verified 14 Sep 2026
BenchmarkInput $1.50 · Output $7.50Campaign rate recorded with the Mistral benchmark runs.Snapshot 15 Oct 2026
- Public release
- API snapshot observed 13 Sep 2026
- Parameters
- Not disclosed
- Open weights
- No
- Architecture / model class
- Closed Medium-tier model; architecture undisclosed
- Reasoning capability / benchmark mode
- Configurable reasoning effort; benchmark: high
- Context window
- 256,000
- Maximum output
- Not specified
- Knowledge / training cutoff
- Not disclosed
Mistral Small 4MistralRequested alias mistral-small-latestResolved model mistral-small-2603+
Latest verified API rateInput $0.15 · Output $0.60Verified 14 Sep 2026
BenchmarkInput $0.15 · Output $0.60Campaign rate recorded with the Mistral benchmark runs.Snapshot 15 Oct 2026
- Public release
- API snapshot observed 13 Sep 2026
- Parameters
- Not disclosed
- Open weights
- No
- Architecture / model class
- Closed Small-tier model; architecture undisclosed
- Reasoning capability / benchmark mode
- Configurable reasoning effort; benchmark: high
- Context window
- 256,000
- Maximum output
- Not specified
- Knowledge / training cutoff
- Not disclosed
Pricing: standard API rates in USD per one million tokens; input is shown before output. Current rates are manually verified, not live. Cached, batch, regional, tax and tool charges are omitted unless a listed exception materially changes the comparison.
* Grok 4.3: 6 May 2026 is the earliest dated official production reference identified. “Not disclosed” indicates that no official value is published. The two V4 Flash rows are separate benchmark configurations of the same model weights.