TL;DR
Luna was cheaper in the August 8, 2026 next-action loop, at $1.67 versus Gemini 3 Flash Preview's $24.95 per 1,000 decisions. Flash Preview answered the 22 stock SQL questions more successfully, 81.8% versus 59.1%, and completed that workflow faster on average. These are separate workload results with different scoring and timing measures; the SQL table supplies no billed-cost comparison or investment-return conclusion.
Next-action execution model bakeoff: published results
These exact August 8, 2026 cost cells belong to the execution replay. Labels are rounded; the full table preserves scores and valid-output rates.
Show the table · 2 rows, 5 columns
Scroll sideways to compare all columns. Select a heading to sort.
| openai/gpt-5.6-luna | 89.2 | 99.0% | $1.67 | 5.6s |
| google/gemini-3-flash-preview | 83.3 | 97.1% | $24.95 | 6.7s |
No matching rows. Clear the filter to see all records.
23-model campaign; 69 frozen decisions × 3 samples; 12 published model rows. Published 2026-08-08.
Which fits the tested job?
Luna cost less in the tested next-action loop; Gemini 3 Flash Preview answered the SQL questions more successfully. The exact version and task matter more here than an OpenAI-versus-Google headline.
Next-action execution model bakeoff
Luna scored 89.2 versus Flash Preview's 83.3, with schema validity 99.0% versus 97.1%. Historical cost was $1.67 versus $24.95 per 1,000 decisions and production p50 5.6s versus 6.7s. Scores remain judge-dependent; these operational results concern the frozen next-action replay.
Models answering 22 stock questions with SQL
Flash Preview averaged 0.791 versus Luna's 0.550, with success 81.8% versus 59.1% and average execution 18.9s versus 27.0s. Prefer testing Flash Preview for this SQL-answer task when correctness and successful answers drive the choice. No billed-cost comparison or investment-return conclusion is available from this table.
Choose the next allowed agent action: follow the published task
A well-formed action can still be the wrong decision. Keep output validity and judged decision quality separate.
Published task summary
The existing plan, conversation and results returned by tools.
Choose one next allowed action, or finish with an answer when the evidence is sufficient.
Read what came back
A structured action. The replay records the proposed action without executing its tools.
Replay 69 saved decisions three times each. Check the required output structure separately from the judge score.
Results from tasks both models attempted
The tested model IDs are openai/gpt-5.6-luna and google/gemini-3-flash-preview. Every result table below comes from a study in which both participated.
Compare the measured quality, historical billed cost and elapsed time within the same task and rubric.
Scores from different tasks are not combined into one winner score.
Next-action execution model bakeoff: read these measurements
Read the complete rubric, sample and billing notes
23-model campaign; 69 frozen decisions × 3 samples; 12 published model rows
Choose one next action in an unfinished trading-agent workflow. This replay measures individual decisions, not a fresh complete trading run.
Scoring
Mean action score comes from the judge. Schema validity is the share of actions matching the required output structure; a valid structure can still contain the wrong action. Switching judges reduced the Luna-versus-DeepSeek score gap from 13.9 to 1.5 points.
Cost units
Cost uses historical provider billing per 1,000 decisions. Reused prefixes earned cache discounts; the cached-token split was not retained.
Time units
Prod p50 is median production response latency, separate from replay quality and complete agent wall time.
Models answering 22 stock questions with SQL: published results
These are exact success-rate cells from 22 questions. Labels are rounded; no billed-cost comparison is reported for this SQL study.
Show the table · 2 rows, 5 columns
Scroll sideways to compare all columns. Select a heading to sort.
| google/gemini-3-flash-preview | 0.791 | 1.00 | 81.8% | 18.9s |
| openai/gpt-5.6-luna | 0.550 | 0.75 | 59.1% | 27.0s |
No matching rows. Clear the filter to see all records.
22 natural-language questions; 6 published model rows. Published 2026-08-08.
Answer a stock question with a data query: follow the published task
This published example explains the task. It is not an identified scored fixture or an answer attributed to any one model. Stored P/E and price divided by the displayed EPS are different fields in its query.
Published worked example, without model attribution
What were the trailing P/E ratios for TSLA and GOOGL on 2025-05-21?
Translate the question into DuckDB SQL, a query over the financial data, then inspect the returned rows and explain the answer.
Read what came back
The original example reported GOOGL at $168.56 with stored P/E 18.46, and TSLA at $334.62 with stored P/E 169.17. Both rows show 2025-05-21.
Inspect both tickers, the requested date and four distinct fiscal quarters of earnings. The original EPS query did not deduplicate fiscal periods, and the correction has not been rerun.
Models answering 22 stock questions with SQL: read these measurements
Read the complete rubric, sample and billing notes
22 natural-language questions; 6 published model rows
Answer financial questions by writing and executing DuckDB SQL against the stock data lake.
Scoring
The average is a blended score across all question cells, including zeros for clarification or execution failure. Success rate and graded-only correctness answer different questions; none measures investment returns.
Cost units
The published screening table has no per-model cost column.
Time units
Average execution time covers the SQL-answer workflow, not the executor study's production p50.
Graded-only mean
The mean among answers that reached grading. Luna recorded 0.931 across 13 graded cells; nine ungraded cells lowered its blended score to 0.550.
TTM earnings / P/E
Trailing twelve-month earnings cover four fiscal quarters. Price-to-earnings relates price to earnings; the published query reads a stored P/E field rather than recomputing it from the displayed EPS.
No comments yet.