TL;DR
Luna Pro had the highest published next-action mean in the August 8, 2026 replay, while Luna was the cost-oriented deployment choice. Luna cost $1.67 per 1,000 decisions with 5.6-second median production latency and 99.0% valid output. Its quality gap with DeepSeek narrowed from 13.9 to 1.5 points under another judge. The 69 decisions, sampled three times each, measure individual actions rather than a complete new trading run.
Published results
23-model campaign; 69 frozen decisions × 3 samples; 12 published model rows. Published 2026-08-08. Values retain their original units and grading scale.
Show the table · 12 rows, 5 columns
Scroll sideways to compare all columns. Select a heading to sort.
| openai/gpt-5.6-luna-pro | 91.7 | 100.0% | $25.05 | 9.9s |
| openai/gpt-5.6-luna | 89.2 | 99.0% | $1.67 | 5.6s |
| meta/muse-spark-1.2 | 86.1 | 99.5% | $44.23 | 7.2s |
| x-ai/grok-build-0.1 | 84.7 | 100.0% | $26.88 | 15.4s |
| google/gemini-3-flash-preview | 83.3 | 97.1% | $24.95 | 6.7s |
| z-ai/glm-5.2 | 82.4 | 98.1% | $25.37 | 15.7s |
| google/gemini-3.6-flash | 80.1 | 100.0% | $24.21 | 7.7s |
| deepseek/deepseek-v4-flash | 75.3 | 98.1% | $6.23 | 17.9s |
| mistralai/mistral-small-2603 | 73.0 | 97.6% | $1.00 | 6.3s |
| google/gemini-3.5-flash-lite | 72.3 | 99.0% | $7.17 | 2.0s |
| nvidia/nemotron-3-ultra-550b | 57.1 | 77.5% | $35.28 | n/a |
| poolside/laguna-xs-2.1 | 51.1 | 66.5% | $5.58 | 4.3s |
No matching rows. Clear the filter to see all records.
23-model campaign; 69 frozen decisions × 3 samples; 12 published model rows. Published 2026-08-08. Values retain their original units and grading scale.
What the result helps you choose
Luna Pro had the highest published mean, 91.7, at $25.05 per 1,000 decisions. Luna scored 89.2 at $1.67 and a 5.6-second production median. The August deployment choice favored Luna's cost and response time; a different judge narrowed its score gap with DeepSeek from 13.9 to 1.5 points.
Choose the next allowed agent action: follow the published task
A well-formed action can still be the wrong decision. Keep output validity and judged decision quality separate.
Published task summary
The existing plan, conversation and results returned by tools.
Choose one next allowed action, or finish with an answer when the evidence is sufficient.
Read what came back
A structured action. The replay records the proposed action without executing its tools.
Replay 69 saved decisions three times each. Check the required output structure separately from the judge score.
Task and tested input
Choose one next action in an unfinished trading-agent workflow. This replay measures individual decisions, not a fresh complete trading run.
Instruction in brief (task summary)
Given the existing plan, conversation and tool results, choose the next allowed action or finish with an answer. Each replay uses frozen messages and the decision request. The article describes this task; the complete private prompts are not published.
Test against your own workload
Build a follow-up evaluation for your workload
Check whether the tested task and required output match your workload. Use a fixed set of representative examples, repeat the runs and inspect failures. Compare cost and latency alongside quality.
Each study lists the publication date and exact model versions. Use those dates when comparing the historical results with a model you can run today.
Calculate the decision from the measured rows
Exact August 8, 2026 billed values from the matched Luna/DeepSeek table, including historical cache effects. This is not a current API price quote or a quality ranking.
Show the table · 2 rows, 5 columns
Scroll sideways to compare all columns. Select a heading to sort.
| openai/gpt-5.6-luna | 89.2 | 99.0% | $1.67 | 5.6s |
| deepseek/deepseek-v4-flash | 75.3 | 98.1% | $6.23 | 17.9s |
No matching rows. Clear the filter to see all records.
Against the prior DeepSeek arm, Luna's published mean increases 13.9 score points and schema validity increases 0.9 percentage points. Recorded billing falls from $6.23 to $1.67 per 1,000 decisions, a 73.2% reduction. Production median latency falls from 17.9s to 5.6s.
The same 69 decisions were sampled three times per arm. Switching judges narrowed the quality gap from 13.9 to 1.5 points, so billing, validity and production latency carry separate evidence. The table preserves the tested historical versions and cache-influenced billing.
How the runs worked
Read the test procedure
23-model campaign; 69 frozen decisions × 3 samples; 12 published model rows
NexusTrade Agent V6 next-action replay
Step 1
Freeze 69 decision inputs. Keep the conversation prefix and decision request identical across model arms.
Step 2
Sample each decision three times. Record the structured action without executing its tools.
Step 3
Check the output contract separately, then grade the proposed action. The public table contains 12 of the 23 campaign arms.
How to read the measurements
Grading and valid output
Mean action score comes from the judge. Schema validity is the share of actions matching the required output structure; a valid structure can still contain the wrong action. Switching judges reduced the Luna-versus-DeepSeek score gap from 13.9 to 1.5 points.
Cost
Cost uses historical provider billing per 1,000 decisions. Reused prefixes earned cache discounts; the cached-token split was not retained.
Time
Prod p50 is median production response latency, separate from replay quality and complete agent wall time.
Sample and scope
Only 12 of the 23 campaign arms are in the public table. The catalog does not fabricate the unpublished rows or their failure counts.
Cross-vendor regrading changed the reported gaps. In the source, Luna versus DeepSeek moved from a 13.9-point score gap to 1.5 points under another judge. Quality claims remain judge-dependent.
Reported costs benefited from reused prompt prefixes. The source did not retain the cached-token split. Historical billed cost is not a current API price quote.
Read the original evidence
Aurora product context
See the broader product workflow
This August recording shows the broader Aurora agent workflow. The task explanations above describe each evaluation; its private conversations and complete system prompts remain private.
No comments yet.