GPT 5.6 Luna: trading task evaluations

For routing and planning, GPT 5.6 Luna recorded a mean score of 0.731 with standard error 0.068. Historical billing was $0.0012 per decision, and median response time was 8.6s. Compare that cost and response time with the other models in the same 29-conversation study; the mean alone does not establish a decisive quality advantage.

No matching rows. Clear the filter to see all records.

16-model campaign, 29 frozen conversations; 14 model rows shown in the public table

Route an incoming request: write a plan, call a catalog tool, ask for clarification or continue an existing agent.

1. Scoring

   Top planning means were close at this sample size. The source's paired comparison with Opus was inconclusive.

2. Cost units

   Cost is provider-billed dollars per decision. Prefix reuse can make it differ from list price multiplied by every input token.

3. Time units

   p50 is median latency in seconds, not the duration of complete research.

4. Frozen conversation

   A saved input kept identical across model arms so their decisions can be compared.

5. Standard error (SE)

   An estimate of uncertainty around the sample mean. A small mean difference needs more evidence than the ranking alone.

For the next allowed action, GPT 5.6 Luna recorded a judge mean of 89.2 and produced valid structured output 99.0% of the time. Historical billing was $1.67 per 1,000 decisions; production median response time was 5.6s. Decision quality, readable output and operating cost answer different questions.

No matching rows. Clear the filter to see all records.

23-model campaign; 69 frozen decisions × 3 samples; 12 published model rows

Choose one next action in an unfinished trading-agent workflow. This replay measures individual decisions, not a fresh complete trading run.

1. Scoring

   Mean action score comes from the judge. Schema validity is the share of actions matching the required output structure; a valid structure can still contain the wrong action. Switching judges reduced the Luna-versus-DeepSeek score gap from 13.9 to 1.5 points.

2. Cost units

   Cost uses historical provider billing per 1,000 decisions. Reused prefixes earned cache discounts; the cached-token split was not retained.

3. Time units

   Prod p50 is median production response latency, separate from replay quality and complete agent wall time.

For artifact review, GPT 5.6 Luna found 83.1% of the hand-labeled defects and repeated its grade on 19 of 23 artifacts. These are separate measurements of finding defects and consistency. Cost and latency are not published for this comparison.

No matching rows. Clear the filter to see all records.

23 hand-labeled artifacts covering 14 task types; 3 reducer models in the public table

Judge defects in delivered research artifacts, including code and validation evidence.

1. Scoring

   Defect recall is the share of hand-labeled defects found. Repeated-grade agreement counts artifacts assigned the same grade on a repeat; agreement alone does not prove correctness.

2. Cost units

   This comparison table has no per-model billing column.

3. Time units

   This table publishes no per-model latency.

4. Mapper / reducer

   Mappers inspect parts of an artifact independently. The reducer combines their findings and assigns the final grade.

On the ship-traffic research task, GPT 5.6 Luna returned the grade sequence D → B+ → D → B+ → D → B+ over 59m, with reported run cost $2.63. Read the whole sequence: a favorable review followed by a failure exposes a repair that did not hold. A missing cost remains missing, and uneven CPU throttling limits timing comparisons.

No matching rows. Clear the filter to see all records.

5 models on the same single research task; real Modal virtual machines

Correlate Strait of Hormuz ship traffic with the next USO trading session.

1. Scoring

   Grades describe the delivered artifact after successive reviews. Oscillation exposed holes in agent-written self-checks; a successful process exit was insufficient.

2. Cost units

   Cost is historical billed run cost. Flash Lite has no reported total after failing out; a dash is missing data, not zero.

3. Time units

   Wall time covers the complete workflow. Uneven web-tier CPU throttling affected timing, so the source treats it as a soft comparison.

4. Sandbox

   An isolated machine where the agent runs research code.

On 22 stock questions, GPT 5.6 Luna recorded a blended score of 0.550, success rate 59.1% and average workflow time 27.0s. The blended score includes zeros for non-answers; it is different from correctness among answers that reached grading. The example below explains the data task without attributing its answer to this model.

No matching rows. Clear the filter to see all records.

22 natural-language questions; 6 published model rows

Answer financial questions by writing and executing DuckDB SQL against the stock data lake.

1. Scoring

   The average is a blended score across all question cells, including zeros for clarification or execution failure. Success rate and graded-only correctness answer different questions; none measures investment returns.

2. Cost units

   The published screening table has no per-model cost column.

3. Time units

   Average execution time covers the SQL-answer workflow, not the executor study's production p50.

4. Graded-only mean

   The mean among answers that reached grading. Luna recorded 0.931 across 13 graded cells; nine ungraded cells lowered its blended score to 0.550.

5. TTM earnings / P/E

   Trailing twelve-month earnings cover four fiscal quarters. Price-to-earnings relates price to earnings; the published query reads a stored P/E field rather than recomputing it from the displayed EPS.