TL;DR
Gemini 3.1 Flash Lite was fast and inexpensive in the August 2026 planning replay, with a 1.8 second median and $0.0031 per decision, but its mean score was 0.586. In the April options task it scored 53/100 with a weak verdict. Planning latency and a complete strategy workflow measure different jobs; creating six portfolio artifacts did not establish that the account-doubling objective was achieved.
- Exact tested ID
- google/gemini-3.1-flash-lite
- Published studies
- 2
- Provider
Eleven models building an options strategy: compare the tested outputs
11 model runs; same task, account context, tools and 25-iteration budget. Published 2026-04-06. Values retain their original units and grading scale. The unchanged table retains the other reported dimensions. These scores describe the tested task rather than investment returns.
Show the table · 2 rows, 6 columns
Scroll sideways to compare all columns. Select a heading to sort.
| Gemini 3.1 Flash Lite | 53 | weak | 0 | 6 | 5.3m |
| Gemini 3 Flash Preview | 66 | mixed | 3 | 6 | 45.5m |
No matching rows. Clear the filter to see all records.
11 model runs; same task, account context, tools and 25-iteration budget. Published 2026-04-06. Values retain their original units and grading scale.
google/gemini-3.1-flash-lite was tested on 2026-04-06. 11 model runs; same task, account context, tools and 25-iteration budget. Its row and the other published arm use the same task and measurement units.
Planning models on 29 frozen conversations: compare the tested outputs
Tick: recorded mean score. Whisker: mean ± one recorded standard error (SE), not a confidence interval. Overlap alone does not establish a ranking.
16-model campaign, 29 frozen conversations; 14 model rows shown in the public table. Published 2026-08-08. Values retain their original units and grading scale.
Show the table · 2 rows, 5 columns
Scroll sideways to compare all columns. Select a heading to sort.
| google/gemini-3.1-flash-lite | 0.586 | 0.066 | $0.0031 | 1.8s |
| meta/muse-spark-1.1 | 0.759 | 0.079 | $0.0229 | 9.6s |
No matching rows. Clear the filter to see all records.
16-model campaign, 29 frozen conversations; 14 model rows shown in the public table. Published 2026-08-08. Values retain their original units and grading scale.
google/gemini-3.1-flash-lite was tested on 2026-08-08. 16-model campaign, 29 frozen conversations; 14 model rows shown in the public table. Its row and the other published arm below use the same task and measurement units.
What this version helps you choose
This version responded quickly in the planning replay, but its lower plan score and weak options verdict make those different jobs separate choices.
Develop an options strategy: follow the published task
A score measures how well the workflow addressed that objective. Portfolio and subagent counts describe activity; the scorecard and narrative disagree on some counts.
The same prompt went to all 11 models
Look at my watchlist and current market conditions, design a profitable options trading strategy that I should use at Monday at open. My goal is to double my $25,000 Public Portfolio live-trading account this year.
The models had the same watchlist, market data, account context, tools and 25-iteration budget.
Flash acted; GPT-5.4 handed the next step back
The published Flash trace fetched market data, launched two specialist agents, read their backtests and called deployPortfolio. GPT-5.4 fetched market data and planned three tracks, then stopped with: "Please proceed by launching those subagents."
The trace demonstrates the difference between choosing the next action and carrying it out. Its Flash process counts disagree with the preserved scorecard.
Read the actual Flash evaluator result
The evaluator returned overallScore 66, verdict mixed and deploymentVerdict iterate_first for Regime-Adaptive Options (NVDA/META). Its listed yearly returns were 2022 +31.22%, 2023 +15.41% and 2024 +0.34%, averaging +15.66%.
The evaluator capped strategy fitness at 3/10 because the reported average was below 30%. It recommended more iteration despite the trace showing a deployment action. Those separate test-year returns do not record achieved account doubling.
Eleven models building an options strategy
Gemini 3.1 Flash Lite scored 53/100 with a weak verdict over 5.3m. The goal was an options strategy for a $25,000 account, with the same tools and 25-iteration budget. The verdict grades progress on that goal; it does not record achieved account doubling.
Show the table · 5 rows, 2 columns
| Published metric | Value |
|---|---|
| Score | 53 |
| Verdict | weak |
| Subagents | 0 |
| Portfolios | 6 |
| Time | 5.3m |
No matching rows. Clear the filter to see all records.
Eleven models building an options strategy: read these measurements
Read the complete rubric, sample and billing notes
11 model runs; same task, account context, tools and 25-iteration budget
Develop an options strategy for a $25,000 account with the stated goal of doubling it within a year.
Scoring
The evaluator weights strategy fitness 40%, evidence 25%, exploration 20% and risk realism 15%. No deployment caps fitness at 2/10; single-year evidence caps it at 4/10. The score is a judgment, not a probability of doubling money.
Cost units
This scorecard has no measured billed-dollar cost column.
Time units
Time includes tool use and historical backtesting, not one response.
Subagent
An agent delegated part of the research to another agent.
Portfolio
A strategy artifact created during the run. Its existence alone does not establish a successful strategy.
Planning models on 29 frozen conversations
For routing and planning, Gemini 3.1 Flash Lite recorded a mean score of 0.586 with standard error 0.066. Historical billing was $0.0031 per decision, and median response time was 1.8s. Compare that cost and response time with the other models in the same 29-conversation study; the mean alone does not establish a decisive quality advantage.
Show the table · 4 rows, 2 columns
| Published metric | Value |
|---|---|
| Score | 0.586 |
| +/- SE | 0.066 |
| $ / decision | $0.0031 |
| p50 | 1.8s |
No matching rows. Clear the filter to see all records.
Route a request and plan the work: follow the published task
A useful planner chooses the right kind of work before doing it. The complete customer conversations and system prompts remain private.
Published task summary
An incoming request and its existing conversation.
Choose whether to write a plan, call a catalog tool, ask for clarification or continue an existing agent.
Read what came back
A routing decision and plan. Every model receives the same saved conversation; the test compares that decision before fresh research begins.
Replay 29 frozen conversations across the model arms, then compare graded plan scores, billing and response time.
Planning models on 29 frozen conversations: read these measurements
Read the complete rubric, sample and billing notes
16-model campaign, 29 frozen conversations; 14 model rows shown in the public table
Route an incoming request: write a plan, call a catalog tool, ask for clarification or continue an existing agent.
Scoring
Top planning means were close at this sample size. The source's paired comparison with Opus was inconclusive.
Cost units
Cost is provider-billed dollars per decision. Prefix reuse can make it differ from list price multiplied by every input token.
Time units
p50 is median latency in seconds, not the duration of complete research.
Frozen conversation
A saved input kept identical across model arms so their decisions can be compared.
Standard error (SE)
An estimate of uncertainty around the sample mean. A small mean difference needs more evidence than the ranking alone.
No comments yet.