Benchmark the router,not just the model.
On October 25, 2026, phantom-1 and each tier's model took the same 295 tasks. Here is how many passed, what each one cost and how long each task took.
- Real-world tasks
- Every tier on the same tasks
- Run it yourself
Routing tiers
- EconomyShort, simple text jobs
- StandardMost coding steps
- FrontierTasks that need the strongest model
phantom-1 picks one of these tiers for each request.
Price your own workload.
Tell it how much you code. phantom-1 sends the hard part to your Frontier model and the rest to cheaper ones.
$178.16saved a month, 66% less
- On phantom-1
- $90.82
- On Claude Sonnet 5.5 alone
- $268.98
50 prompts a day (1,400 model calls) for 30 days at today's prices: 25% on Claude Sonnet 5.5, 75% on DeepSeek V4.1 Flash.
Pick your models. Watch it route.
Choose a model for each tier, edit the request, and see where phantom-1 sends it.
Your lineup
Routed to StandardDeepSeek V4.1 Flash. Typical coding step
Shows where it would go. No model answers it, and nothing is stored.
View the code
from openai import OpenAI
client = OpenAI(base_url="https://phantom.codes/v1", api_key="sk-phantom-...")
response = client.chat.completions.create(
model="phantom-1",
messages=[{"role": "user", "content": "Build a landing page for my app."}],
)
print(response.choices[0].message.content)What the runmeasured.
phantom-1 and each tier's model took the same 295 tasks: how many passed, what they cost, and how long each task took.
phantom-1Router
Picks a tier per requestRuns your lineup
- Pass rate
- 93.9%
- Cost per run
- $0.855
- Median latency
- 2.9s
October 25, 2026295 tasks
EconomyNemotron 3.5 Lightning Free
Short, simple text jobs
nvidia/nemotron-3.5-lightning-free- Pass rate
- 81.7%
- Cost per run
- $0
- Median latency
- 5.5s
StandardDeepSeek V4.1 Flash
Most coding steps
deepseek/deepseek-v4.1-flash- Pass rate
- 90.2%
- Cost per run
- $0.189
- Median latency
- 2.4s
FrontierClaude Sonnet 5.5
Tasks that need the strongest model
anthropic/claude-sonnet-5.5- Pass rate
- 99.3%
- Cost per run
- $3.951
- Median latency
- 2.1s
Quality against cost.
Each point is one configuration: phantom-1, or one tier's model on every task.
- phantom-1
- Economy
- Standard
- Frontier
Quality against latency.
The same configurations, by median seconds per task.
- phantom-1
- Economy
- Standard
- Frontier
Results by task category.
Pass rate for each configuration in each part of the suite the run covered.
- phantom-1
- Economy
- Standard
- Frontier
| Category | phantom-1 | Economy | Standard | Frontier |
|---|---|---|---|---|
| Formatting, 40 tasks | 92.5% | 92.5% | 100% | 100% |
| Small functions, 70 tasks | 87.1% | 94.3% | 87.1% | 97.1% |
| Bug fixing, 40 tasks | 87.5% | 85% | 85% | 100% |
| Algorithms, 45 tasks | 100% | 28.9% | 71.1% | 100% |
| Concurrency, 35 tasks | 100% | 85.7% | 100% | 100% |
| Math, 40 tasks | 100% | 95% | 100% | 100% |
| Long context, 25 tasks | 96% | 92% | 96% | 100% |
Overall leaderboard.
Ranked by pass rate over all 295 tasks; a tie goes to the cheaper run. Latency is the median time per task.
| Rank | Configuration | Pass rate | Cost per run | Median latency |
|---|---|---|---|---|
| 1 | Frontier (Claude Sonnet 5.5) | 99.3% | $3.951 | 2.1s |
| 2 | phantom-1 | 93.9% | $0.855 | 2.9s |
| 3 | Standard (DeepSeek V4.1 Flash) | 90.2% | $0.189 | 2.4s |
| 4 | Economy (Nemotron 3.5 Lightning Free) | 81.7% | $0 | 5.5s |
Where eachtask went.
How often phantom-1 picked each tier across the run's 295 tasks.
Real tasks, real routing decisions.
phantom-1 made these picks when this page was built.
| Task | Task type | Selected model | Reason |
|---|---|---|---|
| Find the remainder when 7^2024 is divided by 1000. | Math | Frontier | Competition math problem |
| Write a function that merges overlapping intervals. | Algorithms | Standard | Typical coding step |
| Extract every email address in this text as JSON. | Formatting | Economy | Routine text work |
| Find the deadlock in this worker pool. | Concurrency | Frontier | Concurrency bug |
| Fix the off-by-one bug in this loop. | Bug fixing | Standard | Typical coding step |
How we ranthe benchmark.
The October 25, 2026 run sent the same 295 tasks to phantom-1 and to each tier's model alone, checked every answer, and recorded what it cost at caller prices and how long it took.
1. Task suite
The base set holds everyday tasks in 7 categories: formatting, small functions, bug fixing, algorithms, concurrency, math, long context.
The hard set, not part of this run, adds public benchmarks:
- LiveCodeBench
- AIME
- BigCodeBench-Hard
- MMLU-Pro
2. Configurations
phantom-1 as callers run it, and each tier's model on every task.
- phantom-1
- Always economy
- Always standard
- Always frontier
3. What a run measures
Each answer is checked, timed and priced.
- Pass rate: tasks passed out of tasks run
- Cost at caller prices, added up over the run
- Latency: the median time per task
Reproduce this run
From a checkout of the Phantom repository, run the tasks, then make the report. The run stops when it reaches the dollar cap you set, and it reads its upstream key from BENCH_GATEWAY_KEY. The report writes the summary this page reads.
pnpm bench:run -- --max-usd 15 pnpm bench:report
Put phantom-1 to the test.
Send phantom-1 as the model. Each response says which model ran.