TL;DR
Luna had lower historical cost, faster median responses and better recorded outcomes than DeepSeek V4 Flash on the August 8, 2026 agent workloads. Next-action cost was $1.67 versus $6.23 per 1,000 decisions. The judged quality gap narrowed from 13.9 to 1.5 points under another judge. On the single sandbox task, Luna reached B+ three times with intervening failures, while DeepSeek received six D grades.
Planning models on 29 frozen conversations: published results
Tick: recorded mean score. Whisker: mean ± one recorded standard error (SE), not a confidence interval. Overlap alone does not establish a ranking.
16-model campaign, 29 frozen conversations; 14 model rows shown in the public table. Published 2026-08-08.
Show the table · 2 rows, 5 columns
Scroll sideways to compare all columns. Select a heading to sort.
| openai/gpt-5.6-luna | 0.731 | 0.068 | $0.0012 | 8.6s |
| deepseek/deepseek-v4-flash | 0.493 | 0.080 | $0.0032 | 12.4s |
No matching rows. Clear the filter to see all records.
16-model campaign, 29 frozen conversations; 14 model rows shown in the public table. Published 2026-08-08.
Which fits the tested job?
Luna is the stronger candidate for the published agent workflow: it had higher recorded planning and execution scores, lower billed cost in both decision studies and better results on the sandbox task. The execution quality gap shrank sharply under a different judge, so the operational cost and schema results matter alongside the score.
Planning models on 29 frozen conversations
Luna scored 0.731 versus DeepSeek's 0.493, at $0.0012 versus $0.0032 per decision. Median response time was 8.6s versus 12.4s. Those observations support trying Luna for this router; the 29-conversation sample does not establish general planning superiority.
Next-action execution model bakeoff
The original judge scored Luna 89.2 and DeepSeek 75.3. Another judge reduced that gap from 13.9 to 1.5 points. Luna still recorded 99.0% versus 98.1% schema validity, $1.67 versus $6.23 per 1,000 decisions and 5.6s versus 17.9s production p50. Judge-sensitive quality and observed operational savings are separate reasons for a choice.
Five models completing one sandbox research task
On the single ship-traffic research task, Luna reached B+ three times; DeepSeek remained at D through six grades. The recorded totals were 59m/$2.63 and 181m/$5.17. Remediation oscillation and uneven CPU throttling prevent treating this as a reliable general coding-speed benchmark.
Route a request and plan the work: follow the published task
A useful planner chooses the right kind of work before doing it. The complete customer conversations and system prompts remain private.
Published task summary
An incoming request and its existing conversation.
Choose whether to write a plan, call a catalog tool, ask for clarification or continue an existing agent.
Read what came back
A routing decision and plan. Every model receives the same saved conversation; the test compares that decision before fresh research begins.
Replay 29 frozen conversations across the model arms, then compare graded plan scores, billing and response time.
Results from tasks both models attempted
The tested model IDs are openai/gpt-5.6-luna and deepseek/deepseek-v4-flash. Every result table below comes from a study in which both participated.
Compare the measured quality, historical billed cost and elapsed time within the same task and rubric.
Scores from different tasks are not combined into one winner score.
Planning models on 29 frozen conversations: read these measurements
Read the complete rubric, sample and billing notes
16-model campaign, 29 frozen conversations; 14 model rows shown in the public table
Route an incoming request: write a plan, call a catalog tool, ask for clarification or continue an existing agent.
Scoring
Top planning means were close at this sample size. The source's paired comparison with Opus was inconclusive.
Cost units
Cost is provider-billed dollars per decision. Prefix reuse can make it differ from list price multiplied by every input token.
Time units
p50 is median latency in seconds, not the duration of complete research.
Frozen conversation
A saved input kept identical across model arms so their decisions can be compared.
Standard error (SE)
An estimate of uncertainty around the sample mean. A small mean difference needs more evidence than the ranking alone.
Next-action execution model bakeoff: published results
These are exact August 8, 2026 execution-table values. The separate judge-score gap depends on the judge, and the sandbox outcome is a different task.
Show the table · 2 rows, 5 columns
Scroll sideways to compare all columns. Select a heading to sort.
| openai/gpt-5.6-luna | 89.2 | 99.0% | $1.67 | 5.6s |
| deepseek/deepseek-v4-flash | 75.3 | 98.1% | $6.23 | 17.9s |
No matching rows. Clear the filter to see all records.
23-model campaign; 69 frozen decisions × 3 samples; 12 published model rows. Published 2026-08-08.
Choose the next allowed agent action: follow the published task
A well-formed action can still be the wrong decision. Keep output validity and judged decision quality separate.
Published task summary
The existing plan, conversation and results returned by tools.
Choose one next allowed action, or finish with an answer when the evidence is sufficient.
Read what came back
A structured action. The replay records the proposed action without executing its tools.
Replay 69 saved decisions three times each. Check the required output structure separately from the judge score.
Next-action execution model bakeoff: read these measurements
Read the complete rubric, sample and billing notes
23-model campaign; 69 frozen decisions × 3 samples; 12 published model rows
Choose one next action in an unfinished trading-agent workflow. This replay measures individual decisions, not a fresh complete trading run.
Scoring
Mean action score comes from the judge. Schema validity is the share of actions matching the required output structure; a valid structure can still contain the wrong action. Switching judges reduced the Luna-versus-DeepSeek score gap from 13.9 to 1.5 points.
Cost units
Cost uses historical provider billing per 1,000 decisions. Reused prefixes earned cache discounts; the cached-token split was not retained.
Time units
Prod p50 is median production response latency, separate from replay quality and complete agent wall time.
Five models completing one sandbox research task: published results
5 models on the same single research task; real Modal virtual machines. Published 2026-08-08.
Show the table · 2 rows, 4 columns
Scroll sideways to compare all columns. Select a heading to sort.
| openai/gpt-5.6-luna | D → B+ → D → B+ → D → B+ | 59m | $2.63 |
| deepseek/deepseek-v4-flash | D → D → D → D → D → D | 181m | $5.17 |
No matching rows. Clear the filter to see all records.
5 models on the same single research task; real Modal virtual machines. Published 2026-08-08.
Write research code and repair its artifacts: follow the published task
The grade sequence shows whether a repair held up at the next review. One B+ followed by D is a different outcome from repeated favorable reviews.
Run one concrete research task
Correlate Strait of Hormuz ship traffic with the next USO trading session.
Write and run research code on real Modal machines, return the artifacts for review and repair defects raised by the grader.
Inspect the recorded repairs and regressions
Luna returned D → B+ → D → B+ → D → B+; GLM returned D → D → B+ → D → D → D. DeepSeek returned D on all six reviews.
The source found self-check code that assigned the same variable to delivered_stats and recomputed, then hard-coded a match. That explains why a passing self-check could survive while the delivered analysis still failed review. The publication does not assign this particular code defect to one model in the table.
Five models completing one sandbox research task: read these measurements
Read the complete rubric, sample and billing notes
5 models on the same single research task; real Modal virtual machines
Correlate Strait of Hormuz ship traffic with the next USO trading session.
Scoring
Grades describe the delivered artifact after successive reviews. Oscillation exposed holes in agent-written self-checks; a successful process exit was insufficient.
Cost units
Cost is historical billed run cost. Flash Lite has no reported total after failing out; a dash is missing data, not zero.
Time units
Wall time covers the complete workflow. Uneven web-tier CPU throttling affected timing, so the source treats it as a soft comparison.
Sandbox
An isolated machine where the agent runs research code.
No comments yet.