TL;DR
GPT 5.6 Luna Pro had the highest published next-action mean, 91.7, with 100% valid structured output in August 2026. It cost $25.05 per 1,000 decisions with 9.9-second production median latency, compared with Luna's 89.2 score, $1.67 cost and 5.6 seconds. Their published planning means both matched at 0.731. The extra next-action score came with a much higher recorded bill; these decision evaluations do not measure complete trading performance.
- Exact tested ID
- openai/gpt-5.6-luna-pro
- Published studies
- 2
- Provider
- openai
Planning models on 29 frozen conversations: compare the tested outputs
Tick: recorded mean score. Whisker: mean ± one recorded standard error (SE), not a confidence interval. Overlap alone does not establish a ranking.
16-model campaign, 29 frozen conversations; 14 model rows shown in the public table. Published 2026-08-08. Values retain their original units and grading scale.
Show the table · 2 rows, 5 columns
Scroll sideways to compare all columns. Select a heading to sort.
| openai/gpt-5.6-luna-pro | 0.731 | 0.068 | $0.0076 | 21.9s |
| meta/muse-spark-1.1 | 0.759 | 0.079 | $0.0229 | 9.6s |
No matching rows. Clear the filter to see all records.
16-model campaign, 29 frozen conversations; 14 model rows shown in the public table. Published 2026-08-08. Values retain their original units and grading scale.
openai/gpt-5.6-luna-pro was tested on 2026-08-08. 16-model campaign, 29 frozen conversations; 14 model rows shown in the public table. Its row and the other published arm use the same task and measurement units.
Next-action execution model bakeoff: compare the tested outputs
August 8, 2026. Luna Pro scored 91.7 with 100.0% validity and 9.9s median latency; Luna scored 89.2 with 99.0% validity and 5.6s latency. Planning means matched at 0.731.
Show the table · 2 rows, 5 columns
Scroll sideways to compare all columns. Select a heading to sort.
| openai/gpt-5.6-luna-pro | 91.7 | 100.0% | $25.05 | 9.9s |
| openai/gpt-5.6-luna | 89.2 | 99.0% | $1.67 | 5.6s |
No matching rows. Clear the filter to see all records.
23-model campaign; 69 frozen decisions × 3 samples; 12 published model rows. Published 2026-08-08. Values retain their original units and grading scale.
openai/gpt-5.6-luna-pro was tested on 2026-08-08. 23-model campaign; 69 frozen decisions × 3 samples; 12 published model rows. Its row and the other published arm below use the same task and measurement units.
What this version helps you choose
Luna Pro had the highest next-action mean. Its recorded bill exceeded Luna’s; on planning their published means matched.
Route a request and plan the work: follow the published task
A useful planner chooses the right kind of work before doing it. The complete customer conversations and system prompts remain private.
Published task summary
An incoming request and its existing conversation.
Choose whether to write a plan, call a catalog tool, ask for clarification or continue an existing agent.
Read what came back
A routing decision and plan. Every model receives the same saved conversation; the test compares that decision before fresh research begins.
Replay 29 frozen conversations across the model arms, then compare graded plan scores, billing and response time.
Planning models on 29 frozen conversations
For routing and planning, GPT 5.6 Luna Pro recorded a mean score of 0.731 with standard error 0.068. Historical billing was $0.0076 per decision, and median response time was 21.9s. Compare that cost and response time with the other models in the same 29-conversation study; the mean alone does not establish a decisive quality advantage.
Show the table · 4 rows, 2 columns
| Published metric | Value |
|---|---|
| Score | 0.731 |
| +/- SE | 0.068 |
| $ / decision | $0.0076 |
| p50 | 21.9s |
No matching rows. Clear the filter to see all records.
Planning models on 29 frozen conversations: read these measurements
Read the complete rubric, sample and billing notes
16-model campaign, 29 frozen conversations; 14 model rows shown in the public table
Route an incoming request: write a plan, call a catalog tool, ask for clarification or continue an existing agent.
Scoring
Top planning means were close at this sample size. The source's paired comparison with Opus was inconclusive.
Cost units
Cost is provider-billed dollars per decision. Prefix reuse can make it differ from list price multiplied by every input token.
Time units
p50 is median latency in seconds, not the duration of complete research.
Frozen conversation
A saved input kept identical across model arms so their decisions can be compared.
Standard error (SE)
An estimate of uncertainty around the sample mean. A small mean difference needs more evidence than the ranking alone.
Next-action execution model bakeoff
For the next allowed action, GPT 5.6 Luna Pro recorded a judge mean of 91.7 and produced valid structured output 100.0% of the time. Historical billing was $25.05 per 1,000 decisions; production median response time was 9.9s. Decision quality, readable output and operating cost answer different questions.
Show the table · 4 rows, 2 columns
| Published metric | Value |
|---|---|
| Mean score | 91.7 |
| Schema valid | 100.0% |
| $ / 1k decisions | $25.05 |
| Prod p50 | 9.9s |
No matching rows. Clear the filter to see all records.
Choose the next allowed agent action: follow the published task
A well-formed action can still be the wrong decision. Keep output validity and judged decision quality separate.
Published task summary
The existing plan, conversation and results returned by tools.
Choose one next allowed action, or finish with an answer when the evidence is sufficient.
Read what came back
A structured action. The replay records the proposed action without executing its tools.
Replay 69 saved decisions three times each. Check the required output structure separately from the judge score.
Next-action execution model bakeoff: read these measurements
Read the complete rubric, sample and billing notes
23-model campaign; 69 frozen decisions × 3 samples; 12 published model rows
Choose one next action in an unfinished trading-agent workflow. This replay measures individual decisions, not a fresh complete trading run.
Scoring
Mean action score comes from the judge. Schema validity is the share of actions matching the required output structure; a valid structure can still contain the wrong action. Switching judges reduced the Luna-versus-DeepSeek score gap from 13.9 to 1.5 points.
Cost units
Cost uses historical provider billing per 1,000 decisions. Reused prefixes earned cache discounts; the cached-token split was not retained.
Time units
Prod p50 is median production response latency, separate from replay quality and complete agent wall time.
No comments yet.