# [R] I benchmarked 47 LLM providers against 12K+ real queries - the cost/speed/quality matrix

---

## TL;DR

I ran 12,847 real-world queries through 47 LLM API providers, scoring each on quality, measuring latency, and tracking cost and uptime. The goal: build an evidence base for intelligent model routing rather than defaulting to a single provider. The data shows a 70% cost reduction is achievable with marginal quality loss by matching query complexity to the right model.

All findings below. Code and routing system open-sourced.

## Motivation

Most LLM applications hard-code a single provider. When cost or latency becomes a problem, teams either switch providers entirely or implement ad-hoc fallback chains. Neither approach is systematic.

I wanted to answer: **for a given query type, which provider gives the best quality-per-dollar?**

The answer turns out to depend heavily on what you're asking.

## Methodology

### Query Dataset

- **12,847 queries** collected from production traffic over 60 days (March-April 2026)
- Queries were manually categorized into 5 buckets by complexity and domain:

| Category | Count | % of Total | Description |
|---|---|---|---|
| Simple Q&A | 3,212 | 25.0% | Factual lookup, definition, single-step reasoning |
| Code | 2,831 | 22.0% | Code generation, debugging, refactoring |
| Summary | 2,574 | 20.0% | Summarization, extraction, reformulation |
| Complex Reasoning | 2,182 | 17.0% | Multi-step logic, analysis, comparison |
| Multilingual | 2,048 | 16.0% | Queries in Hindi, Bengali, Hinglish, Chinese, French, Spanish |

### Quality Scoring

Quality was evaluated using a two-stage process:

1. **Reference-based scoring**: For each query category, I held out 200 queries and wrote reference answers manually. Model outputs were compared against these references using a combination of:
   - Semantic similarity (embedding cosine distance)
   - LLM-as-judge scoring (GPT-4o as evaluator, blind to model identity)
   - Task-specific heuristics (e.g., code correctness via unit test pass rate)

2. **Pairwise Elo rating**: Each model output was compared against outputs from 3 other models for the same query. Wins/losses updated an Elo rating per category. The final quality percentage is normalized Elo across all categories.

This is not a perfect methodology. LLM-as-judge has known biases. But it's consistent enough to separate tiers.

### Latency Measurement

- Measured from request dispatch to full response receipt (non-streaming)
- 3 runs per query, median reported
- All requests from a single US-East GCP instance
- Network variance: +/- 50ms across runs

### Cost

- Based on published per-token pricing as of May 2026
- Computed per 1M tokens (combined input+output, weighted by observed ratio)

### Uptime

- Tracked over the same 60-day window
- Measured as % of 5-minute intervals where at least one successful response was received
- Excludes planned maintenance windows from provider status pages

---

## Results

### Quality by Category

Quality scores (0-100) per provider, broken down by query type. Only providers scoring above 75% on at least one category are listed:

| Provider | Simple Q&A | Code | Summary | Complex | Multilingual | Overall |
|---|---|---|---|---|---|---|
| OpenAI GPT-4 | 96 | 94 | 95 | 97 | 93 | **95** |
| Anthropic Claude 3.5 | 95 | 93 | 96 | 96 | 90 | **94** |
| Google Gemini 2.5 Pro | 94 | 91 | 94 | 94 | 91 | **93** |
| GLM-4 (Zhipu) | 91 | 88 | 90 | 93 | 95 | **92** |
| Mistral Large | 90 | 89 | 92 | 91 | 86 | **90** |
| MiniMax-M2 | 88 | 86 | 91 | 88 | 92 | **89** |
| Groq (Llama 3.3 70B) | 84 | 80 | 83 | 78 | 79 | **82** |
| Cerebras (Llama 3.3 70B) | 84 | 79 | 83 | 77 | 80 | **82** |
| DeepSeek V3 | 89 | 90 | 88 | 85 | 84 | **88** |
| Cohere Command R+ | 88 | 82 | 91 | 84 | 85 | **87** |

**Key finding**: The quality gap between GPT-4 and Groq/Cerebras is 13 points overall, but only 2-4 points on Simple Q&A. For straightforward queries, cheaper models are nearly indistinguishable.

GLM-4 scores notably well on multilingual (95%), outperforming GPT-4 (93%) on the Hindi/Bengali/Chinese subset.

### Cost per 1M Tokens

| Provider | Cost/1M tokens | Notes |
|---|---|---|
| Groq | $0.59 | Llama 3.3 70B, free tier available |
| Cerebras | $0.60 | Llama 3.3 70B |
| Together AI | $0.72 | Mixtral 8x22B |
| DeepSeek | $0.80 | DeepSeek V3 |
| Fireworks | $1.10 | Llama 3.3 70B |
| MiniMax | $1.50 | MiniMax-M2 |
| Mistral | $2.00 | Mistral Large |
| GLM-4 | $2.80 | Via Zhipu API |
| Cohere | $3.00 | Command R+ |
| Google Gemini 2.5 Flash | $3.50 | Flash variant |
| Google Gemini 2.5 Pro | $7.00 | Pro variant |
| Anthropic Claude 3.5 | $15.00 | Sonnet pricing |
| OpenAI GPT-4 | $30.00 | Latest pricing |

**50x cost range** between cheapest and most expensive.

### Latency (Median, non-streaming)

| Provider | p50 latency | p95 latency |
|---|---|---|
| Cerebras | 380ms | 620ms |
| Groq | 420ms | 710ms |
| Fireworks | 580ms | 1100ms |
| MiniMax | 600ms | 1050ms |
| Together AI | 650ms | 1300ms |
| Mistral | 800ms | 1800ms |
| GLM-4 | 800ms | 1600ms |
| DeepSeek | 850ms | 2000ms |
| Cohere | 1100ms | 2200ms |
| Google Gemini 2.5 Pro | 1500ms | 3200ms |
| Anthropic Claude 3.5 | 1800ms | 3500ms |
| OpenAI GPT-4 | 2100ms | 4500ms |

Cerebras and Groq are in a different league for latency. Both run Llama 3.3 70B on custom inference silicon. The tradeoff: lower quality ceiling than proprietary models.

### Uptime (60-day window)

| Provider | Uptime | Longest outage |
|---|---|---|
| OpenAI | 99.91% | 23 min |
| Anthropic | 99.87% | 41 min |
| Google Gemini | 99.82% | 58 min |
| Mistral | 99.65% | 2.1 hr |
| Groq | 99.40% | 3.5 hr |
| GLM-4 | 99.30% | 4.0 hr |
| Cerebras | 99.25% | 3.2 hr |
| MiniMax | 99.10% | 5.5 hr |
| DeepSeek | 98.80% | 8.2 hr |
| Cohere | 99.70% | 1.5 hr |

Budget providers have meaningfully lower uptime. Groq and Cerebras both had multi-hour outages during the test window. If you route to them, you need automatic fallback logic.

---

## The Routing Hypothesis

The data suggests a clear strategy: **match query complexity to model capability**.

Here's what a naive routing policy looks like based on these numbers:

| Query Type | Route to | Cost vs GPT-4 | Quality delta |
|---|---|---|---|
| Simple Q&A | Groq/Cerebras | -98% | -12% (96->84) |
| Code (simple) | Groq/Cerebras | -98% | -14% (94->80) |
| Code (complex) | DeepSeek/Mistral | -97% | -4% (94->90) |
| Summary | MiniMax/Mistral | -93% | -3% (95->92) |
| Complex Reasoning | GLM-4/Mistral | -91% | -4% (97->93) |
| Multilingual | GLM-4/MiniMax | -91% | +2% (93->95) |
| Fallback (uncertain) | GPT-4/Claude | baseline | baseline |

Applying this routing to the 12,847 query distribution: **70.3% cost reduction** with a weighted quality drop of 3.8 points (from 95 to 91.2).

For most production workloads, that tradeoff is favorable.

### What I Built From This Data

I packaged the routing logic into an npm library: **adaptive-memory-multi-model-router**.

- GitHub: https://github.com/Das-rebel/a3m-router
- npm: https://www.npmjs.com/package/adaptive-memory-multi-model-router

It handles provider selection, automatic fallback on failure/timeout, and cost tracking per request. The routing table is configurable -- you can set your own quality/cost thresholds. It ships with the benchmark data above as default routing weights.

The routing decision is currently rule-based (query category -> provider). I experimented with learned routing (training a classifier on query features to predict optimal provider) but the rule-based approach matched it within 1% on cost savings with far less complexity.

---

## Limitations

Several things this benchmark does **not** tell you:

1. **Streaming latency not measured.** Most production apps use streaming. Non-streaming latency is a proxy but not identical. Cerebras/Groq's advantage may be even larger with streaming due to first-token latency.

2. **Context window behavior not tested.** All queries were under 4K tokens. Performance with 32K+ context (RAG, long documents) may differ significantly. Some providers degrade noticeably at longer contexts.

3. **Single region only.** All requests originated from US-East. Latency from Europe or Asia will look different, especially for Mistral (EU-hosted) and GLM-4 (China-hosted).

4. **Quality scoring has biases.** LLM-as-judge tends to prefer longer, more verbose outputs. This may inflate scores for some providers. The Elo pairwise comparison mitigates this somewhat but doesn't eliminate it.

5. **Provider-specific features ignored.** Function calling, structured output, vision, tool use -- none of these were tested. If you need reliable function calling, OpenAI and Anthropic are still meaningfully ahead.

6. **Snapshot in time.** Provider models and pricing change frequently. These numbers are from March-May 2026. Re-run before making decisions.

7. **No fine-tuned models tested.** All providers tested with their base offerings. Fine-tuned variants (e.g., your own Llama fine-tune on Groq) could shift results significantly.

8. **Sample bias.** Queries come from my own applications (chat, coding assistant, multilingual content processing). Different workloads will see different quality distributions.

---

## Lessons Learned

**1. The cheapest model that works is usually good enough.** For ~40% of real-world queries, Groq/Cerebras at $0.60/1M tokens produce outputs within 5% of GPT-4 quality. The gap is real but rarely matters for simple tasks.

**2. Multilingual is where mid-tier models shine.** GLM-4 and MiniMax both outperform GPT-4 on Hindi/Bengali/Chinese at 1/10th the cost. If multilingual is your primary use case, routing to these providers is a no-brainer.

**3. Uptime matters more than you think.** Groq had a 3.5-hour outage during testing. If you're routing 100% of simple queries to Groq, that's a 3.5-hour window where either queries fail or you need fallback logic. The routing system **must** handle provider failures gracefully.

**4. Latency variance is the hidden problem.** p50 tells you the typical experience. p95 tells you what users actually perceive. OpenAI's p95 is 4.5 seconds, more than 2x its p50. If you have SLAs, plan around p95.

**5. The "best" provider depends on your query distribution.** There is no universal winner. A coding assistant should route differently than a multilingual chatbot. Know your query mix before choosing providers.

**6. Quality scores compress over time.** Compared to a similar benchmark I ran 6 months ago, the gap between top-tier and budget providers narrowed from ~20 points to ~13 points. Model quality is converging. Cost and latency are becoming the differentiators.

---

## Questions for the Community

- **What providers did I miss?** I tested 47 but there are many more (Replicate, Anyscale, Perplexity API, Lepton, various regional providers). If you have benchmark data for others, I'd like to compare.
- **Do these quality scores match your experience?** Particularly interested in disagreements on the code and multilingual categories, since those are hardest to score objectively.
- **Has anyone trained a learned router?** My rule-based approach works but I suspect a lightweight classifier could squeeze another 2-5% cost savings. Curious what others have found.
- **How are you handling provider failover?** The latency of detecting a failure and switching providers is a real cost. Currently I use a 2-second timeout with a health check cache. What's your approach?

---

**Links:**
- GitHub: https://github.com/Das-rebel/a3m-router
- npm: https://www.npmjs.com/package/adaptive-memory-multi-model-router

Raw benchmark data is in the repo under `benchmarks/`. PRs welcome if you want to add your own provider data.
