TL;DR
In the August 8, 2026 planning replay, Muse Spark 1.1 had the highest raw mean score at 0.759; Luna scored 0.731 at $0.0012 per decision versus Muse's $0.0229. The published choice favored cost among close planning scores, not a proven universal quality winner. The study used 29 frozen conversations, with an inconclusive paired comparison against Opus. Treat costs and latency as historical measurements and repeat representative tasks before applying the choice to a different workload or model version.
Published results
Tick: recorded mean score. Whisker: mean ± one recorded standard error (SE), not a confidence interval. Overlap alone does not establish a ranking.
Showing 12 of 14 recorded rows. The complete table is below.
16-model campaign, 29 frozen conversations; 14 model rows shown in the public table. Published 2026-08-08. Values retain their original units and grading scale.
Show the table · 14 rows, 5 columns
Scroll sideways to compare all columns. Select a heading to sort.
| meta/muse-spark-1.1 | 0.759 | 0.079 | $0.0229 | 9.6s |
| openai/gpt-5.6-luna | 0.731 | 0.068 | $0.0012 | 8.6s |
| openai/gpt-5.6-luna-pro | 0.731 | 0.068 | $0.0076 | 21.9s |
| moonshotai/kimi-k3 | 0.731 | 0.068 | $0.0799 | 38.9s |
| x-ai/grok-4.5 | 0.724 | 0.065 | $0.0519 | 23.5s |
| deepseek/deepseek-v4-pro | 0.710 | 0.064 | $0.0142 | 27.2s |
| google/gemini-3.6-flash | 0.662 | 0.070 | $0.0387 | 8.5s |
| anthropic/claude-opus-5 | 0.662 | 0.074 | $0.3159 | 26.6s |
| openai/gpt-5.6-terra-pro | 0.634 | 0.073 | $0.0646 | 16.0s |
| google/gemini-3.1-flash-lite | 0.586 | 0.066 | $0.0031 | 1.8s |
| anthropic/claude-sonnet-5 | 0.579 | 0.071 | $0.1348 | 24.9s |
| z-ai/glm-5.2 | 0.515 | 0.079 | $0.0246 | 44.8s |
| deepseek/deepseek-v4-flash | 0.493 | 0.080 | $0.0032 | 12.4s |
| qwen/qwen3.8-max | 0.400 | 0.082 | $0.0421 | 168.7s |
No matching rows. Clear the filter to see all records.
16-model campaign, 29 frozen conversations; 14 model rows shown in the public table. Published 2026-08-08. Values retain their original units and grading scale.
What the result helps you choose
Muse Spark 1.1 had the highest mean score, 0.759. Luna scored 0.731 at $0.0012 per decision versus Muse's $0.0229. The published choice favored lower recorded cost among close planning scores; the paired comparison with Opus was inconclusive.
Route a request and plan the work: follow the published task
A useful planner chooses the right kind of work before doing it. The complete customer conversations and system prompts remain private.
Published task summary
An incoming request and its existing conversation.
Choose whether to write a plan, call a catalog tool, ask for clarification or continue an existing agent.
Read what came back
A routing decision and plan. Every model receives the same saved conversation; the test compares that decision before fresh research begins.
Replay 29 frozen conversations across the model arms, then compare graded plan scores, billing and response time.
Task and tested input
Route an incoming request: write a plan, call a catalog tool, ask for clarification or continue an existing agent.
Instruction in brief (task summary)
Read the request and conversation, then choose the route and plan. This describes Router V5's task; the 29 frozen customer conversations and complete prompts are not reproduced here.
Test against your own workload
Build a follow-up evaluation for your workload
Check whether the tested task and required output match your workload. Use a fixed set of representative examples, repeat the runs and inspect failures. Compare cost and latency alongside quality.
Each study lists the publication date and exact model versions. Use those dates when comparing the historical results with a model you can run today.
Calculate the decision from the measured rows
August 8, 2026, 29 frozen conversations. Luna mean 0.731 with SE 0.068; Muse mean 0.759 with SE 0.079. Close raw means and the inconclusive paired comparison do not establish a dependable quality winner.
Show the table · 2 rows, 5 columns
Scroll sideways to compare all columns. Select a heading to sort.
| openai/gpt-5.6-luna | 0.731 | 0.068 | $0.0012 | 8.6s |
| meta/muse-spark-1.1 | 0.759 | 0.079 | $0.0229 | 9.6s |
No matching rows. Clear the filter to see all records.
Luna's recorded $0.0012 per decision is 94.76% lower than Muse's $0.0229. The raw mean-score gap is 0.028, while their reported standard errors are 0.068 and 0.079. Read the source's paired comparison before treating the raw ordering as a dependable quality difference.
The published choice favored cost among similar-quality planning candidates. Repeat the same request corpus and grading contract when evaluating a new version; a new task needs its own decision evidence.
How the runs worked
Read the test procedure
16-model campaign, 29 frozen conversations; 14 model rows shown in the public table
NexusTrade Router V5 plan replay
Step 1
Freeze the same 29 production conversation inputs for every arm.
Step 2
Replay the identical corpus across 16 model arms.
Step 3
Compare mean score and standard error, then billing and latency. The source shows 14 rows.
How to read the measurements
Grading and valid output
Top planning means were close at this sample size. The source's paired comparison with Opus was inconclusive.
Cost
Cost is provider-billed dollars per decision. Prefix reuse can make it differ from list price multiplied by every input token.
Time
p50 is median latency in seconds, not the duration of complete research.
Frozen conversation
A saved input kept identical across model arms so their decisions can be compared.
Standard error (SE)
An estimate of uncertainty around the sample mean. A small mean difference needs more evidence than the ranking alone.
Sample and scope
The public table omits two poolside arms for space. This snapshot contains the 14 published rows, not a reconstruction of their private outputs.
The top raw score is Muse Spark 1.1 at 0.759. GPT 5.6 Luna scored 0.731 at a reported $0.0012 per decision; the published selection favored cost among similar-quality candidates, rather than claiming the highest score.
The source reports a paired comparison with Opus as inconclusive. This page does not turn small raw gaps into a confident quality winner.
Read the original evidence
Aurora product context
See the broader product workflow
This August recording shows the broader Aurora agent workflow. The task explanations above describe each evaluation; its private conversations and complete system prompts remain private.
No comments yet.