TL;DR
Gemini 3 Flash Preview led the April 6, 2026 options scorecard with 66/100 and a mixed verdict, rather than achieved account doubling. Its August SQL success rate was 81.8% across 22 questions, higher than Luna's in that comparison. Next-action decisions had 6.7-second median production latency, while Luna cost less. These studies measure different tasks and should guide the choice for the specific job.
- Exact tested ID
- google/gemini-3-flash-preview
- Published studies
- 3
- Provider
Eleven models building an options strategy: compare the tested outputs
These are exact April 6, 2026 scorecard cells. Flash Preview was graded mixed and Flash Lite weak. The scores evaluate progress toward the account goal; they are not achieved account returns.
Show the table · 2 rows, 6 columns
Scroll sideways to compare all columns. Select a heading to sort.
| Gemini 3 Flash Preview | 66 | mixed | 3 | 6 | 45.5m |
| Gemini 3.1 Flash Lite | 53 | weak | 0 | 6 | 5.3m |
No matching rows. Clear the filter to see all records.
11 model runs; same task, account context, tools and 25-iteration budget. Published 2026-04-06. Values retain their original units and grading scale.
google/gemini-3-flash-preview was tested on 2026-04-06. 11 model runs; same task, account context, tools and 25-iteration budget. Its row and the other published arm use the same task and measurement units.
Next-action execution model bakeoff: compare the tested outputs
23-model campaign; 69 frozen decisions × 3 samples; 12 published model rows. Published 2026-08-08. Values retain their original units and grading scale.
Show the table · 2 rows, 5 columns
Scroll sideways to compare all columns. Select a heading to sort.
| google/gemini-3-flash-preview | 83.3 | 97.1% | $24.95 | 6.7s |
| openai/gpt-5.6-luna-pro | 91.7 | 100.0% | $25.05 | 9.9s |
No matching rows. Clear the filter to see all records.
23-model campaign; 69 frozen decisions × 3 samples; 12 published model rows. Published 2026-08-08. Values retain their original units and grading scale.
google/gemini-3-flash-preview was tested on 2026-08-08. 23-model campaign; 69 frozen decisions × 3 samples; 12 published model rows. Its row and the other published arm below use the same task and measurement units.
Models answering 22 stock questions with SQL: compare the tested outputs
Original units preserved. Average score is the historical blended measure; this table is not a new run.
Show the table · 2 rows, 5 columns
Scroll sideways to compare all columns. Select a heading to sort.
| google/gemini-3-flash-preview | 0.791 | 1.00 | 81.8% | 18.9s |
| google/gemini-3.6-flash | 0.841 | 1.00 | 86.4% | 26.6s |
No matching rows. Clear the filter to see all records.
Original units preserved. Average score is the historical blended measure; this table is not a new run.
google/gemini-3-flash-preview was tested on 2026-08-08. 22 natural-language questions; 6 published model rows. Its row and the other published arm below use the same task and measurement units.
What this version helps you choose
Flash Preview led the April options scorecard with a mixed verdict and answered August’s SQL questions more successfully than Luna. Luna cost less in next-action decisions.
Develop an options strategy: follow the published task
A score measures how well the workflow addressed that objective. Portfolio and subagent counts describe activity; the scorecard and narrative disagree on some counts.
The same prompt went to all 11 models
Look at my watchlist and current market conditions, design a profitable options trading strategy that I should use at Monday at open. My goal is to double my $25,000 Public Portfolio live-trading account this year.
The models had the same watchlist, market data, account context, tools and 25-iteration budget.
Flash acted; GPT-5.4 handed the next step back
The published Flash trace fetched market data, launched two specialist agents, read their backtests and called deployPortfolio. GPT-5.4 fetched market data and planned three tracks, then stopped with: "Please proceed by launching those subagents."
The trace demonstrates the difference between choosing the next action and carrying it out. Its Flash process counts disagree with the preserved scorecard.
Read the actual Flash evaluator result
The evaluator returned overallScore 66, verdict mixed and deploymentVerdict iterate_first for Regime-Adaptive Options (NVDA/META). Its listed yearly returns were 2022 +31.22%, 2023 +15.41% and 2024 +0.34%, averaging +15.66%.
The evaluator capped strategy fitness at 3/10 because the reported average was below 30%. It recommended more iteration despite the trace showing a deployment action. Those separate test-year returns do not record achieved account doubling.
Eleven models building an options strategy
Inspect the original single-model metrics
Gemini 3 Flash Preview scored 66/100 with a mixed verdict over 45.5m. The goal was an options strategy for a $25,000 account, with the same tools and 25-iteration budget. The verdict grades progress on that goal; it does not record achieved account doubling.
| Published metric | Value |
|---|---|
| Score | 66 |
| Verdict | mixed |
| Subagents | 3 |
| Portfolios | 6 |
| Time | 45.5m |
No matching rows. Clear the filter to see all records.
Eleven models building an options strategy: read these measurements
Read the complete rubric, sample and billing notes
11 model runs; same task, account context, tools and 25-iteration budget
Develop an options strategy for a $25,000 account with the stated goal of doubling it within a year.
Scoring
The evaluator weights strategy fitness 40%, evidence 25%, exploration 20% and risk realism 15%. No deployment caps fitness at 2/10; single-year evidence caps it at 4/10. The score is a judgment, not a probability of doubling money.
Cost units
This scorecard has no measured billed-dollar cost column.
Time units
Time includes tool use and historical backtesting, not one response.
Subagent
An agent delegated part of the research to another agent.
Portfolio
A strategy artifact created during the run. Its existence alone does not establish a successful strategy.
Next-action execution model bakeoff
Inspect the original single-model metrics
For the next allowed action, Gemini 3 Flash Preview recorded a judge mean of 83.3 and produced valid structured output 97.1% of the time. Historical billing was $24.95 per 1,000 decisions; production median response time was 6.7s. Decision quality, readable output and operating cost answer different questions.
| Published metric | Value |
|---|---|
| Mean score | 83.3 |
| Schema valid | 97.1% |
| $ / 1k decisions | $24.95 |
| Prod p50 | 6.7s |
No matching rows. Clear the filter to see all records.
Choose the next allowed agent action: follow the published task
A well-formed action can still be the wrong decision. Keep output validity and judged decision quality separate.
Published task summary
The existing plan, conversation and results returned by tools.
Choose one next allowed action, or finish with an answer when the evidence is sufficient.
Read what came back
A structured action. The replay records the proposed action without executing its tools.
Replay 69 saved decisions three times each. Check the required output structure separately from the judge score.
Next-action execution model bakeoff: read these measurements
Read the complete rubric, sample and billing notes
23-model campaign; 69 frozen decisions × 3 samples; 12 published model rows
Choose one next action in an unfinished trading-agent workflow. This replay measures individual decisions, not a fresh complete trading run.
Scoring
Mean action score comes from the judge. Schema validity is the share of actions matching the required output structure; a valid structure can still contain the wrong action. Switching judges reduced the Luna-versus-DeepSeek score gap from 13.9 to 1.5 points.
Cost units
Cost uses historical provider billing per 1,000 decisions. Reused prefixes earned cache discounts; the cached-token split was not retained.
Time units
Prod p50 is median production response latency, separate from replay quality and complete agent wall time.
Models answering 22 stock questions with SQL
Inspect the original single-model metrics
On 22 stock questions, Gemini 3 Flash Preview recorded a blended score of 0.791, success rate 81.8% and average workflow time 18.9s. The blended score includes zeros for non-answers; it is different from correctness among answers that reached grading. The example below explains the data task without attributing its answer to this model.
| Published metric | Value |
|---|---|
| Average score | 0.791 |
| Median | 1.00 |
| Success rate | 81.8% |
| Avg execution | 18.9s |
No matching rows. Clear the filter to see all records.
Answer a stock question with a data query: follow the published task
This published example explains the task. It is not an identified scored fixture or an answer attributed to any one model. Stored P/E and price divided by the displayed EPS are different fields in its query.
Published worked example, without model attribution
What were the trailing P/E ratios for TSLA and GOOGL on 2025-05-21?
Translate the question into DuckDB SQL, a query over the financial data, then inspect the returned rows and explain the answer.
Read what came back
The original example reported GOOGL at $168.56 with stored P/E 18.46, and TSLA at $334.62 with stored P/E 169.17. Both rows show 2025-05-21.
Inspect both tickers, the requested date and four distinct fiscal quarters of earnings. The original EPS query did not deduplicate fiscal periods, and the correction has not been rerun.
Models answering 22 stock questions with SQL: read these measurements
Read the complete rubric, sample and billing notes
22 natural-language questions; 6 published model rows
Answer financial questions by writing and executing DuckDB SQL against the stock data lake.
Scoring
The average is a blended score across all question cells, including zeros for clarification or execution failure. Success rate and graded-only correctness answer different questions; none measures investment returns.
Cost units
The published screening table has no per-model cost column.
Time units
Average execution time covers the SQL-answer workflow, not the executor study's production p50.
Graded-only mean
The mean among answers that reached grading. Luna recorded 0.931 across 13 graded cells; nine ungraded cells lowered its blended score to 0.550.
TTM earnings / P/E
Trailing twelve-month earnings cover four fiscal quarters. Price-to-earnings relates price to earnings; the published query reads a stored P/E field rather than recomputing it from the displayed EPS.
No comments yet.