TL;DR
Gemini 3.5 Flash Lite's August 2026 next-action evaluation scored 72.3, with 99.0% valid structured output, $7.17 per 1,000 decisions and 2.0-second production median latency. That speed did not transfer to its sandbox research task, which received D and failed out after 24 minutes. On 22 stock-screening questions it scored 0.664 with 72.7% success. These are different tasks; valid output, workflow success and investment returns are separate measures.
- Exact tested ID
- google/gemini-3.5-flash-lite
- Published studies
- 3
- Provider
Next-action execution model bakeoff: compare the tested outputs
69 frozen decisions with three samples each, published August 8, 2026. These judge scores are separate from structured-output validity and the failed sandbox workflow.
Show the table · 2 rows, 5 columns
Scroll sideways to compare all columns. Select a heading to sort.
| google/gemini-3.5-flash-lite | 72.3 | 99.0% | $7.17 | 2.0s |
| openai/gpt-5.6-luna-pro | 91.7 | 100.0% | $25.05 | 9.9s |
No matching rows. Clear the filter to see all records.
23-model campaign; 69 frozen decisions × 3 samples; 12 published model rows. Published 2026-08-08. Values retain their original units and grading scale.
google/gemini-3.5-flash-lite was tested on 2026-08-08. 23-model campaign; 69 frozen decisions × 3 samples; 12 published model rows. Its row and the other published arm use the same task and measurement units.
Five models completing one sandbox research task: compare the tested outputs
google/gemini-3.5-flash-lite was tested on 2026-08-08. 5 models on the same single research task; real Modal virtual machines. Its row and the other published arm below use the same task and measurement units.
Show the table · 2 rows, 4 columns
Scroll sideways to compare all columns. Select a heading to sort.
| google/gemini-3.5-flash-lite | D, then failed out | 24m | — |
| openai/gpt-5.6-luna | D → B+ → D → B+ → D → B+ | 59m | $2.63 |
No matching rows. Clear the filter to see all records.
5 models on the same single research task; real Modal virtual machines. Published 2026-08-08. Values retain their original units and grading scale.
Models answering 22 stock questions with SQL: compare the tested outputs
Original units preserved. Average score is the historical blended measure; this table is not a new run.
Show the table · 2 rows, 5 columns
Scroll sideways to compare all columns. Select a heading to sort.
| google/gemini-3.5-flash-lite | 0.664 | 0.90 | 72.7% | 19.4s |
| google/gemini-3.6-flash | 0.841 | 1.00 | 86.4% | 26.6s |
No matching rows. Clear the filter to see all records.
Original units preserved. Average score is the historical blended measure; this table is not a new run.
google/gemini-3.5-flash-lite was tested on 2026-08-08. 22 natural-language questions; 6 published model rows. Its row and the other published arm below use the same task and measurement units.
What this version helps you choose
A short response time on next-action decisions did not transfer into a successful sandbox run; that run failed out. Its SQL result belongs to a third task.
Choose the next allowed agent action: follow the published task
A well-formed action can still be the wrong decision. Keep output validity and judged decision quality separate.
Published task summary
The existing plan, conversation and results returned by tools.
Choose one next allowed action, or finish with an answer when the evidence is sufficient.
Read what came back
A structured action. The replay records the proposed action without executing its tools.
Replay 69 saved decisions three times each. Check the required output structure separately from the judge score.
Next-action execution model bakeoff
For the next allowed action, Gemini 3.5 Flash Lite recorded a judge mean of 72.3 and produced valid structured output 99.0% of the time. Historical billing was $7.17 per 1,000 decisions; production median response time was 2.0s. Decision quality, readable output and operating cost answer different questions.
Show the table · 4 rows, 2 columns
| Published metric | Value |
|---|---|
| Mean score | 72.3 |
| Schema valid | 99.0% |
| $ / 1k decisions | $7.17 |
| Prod p50 | 2.0s |
No matching rows. Clear the filter to see all records.
Next-action execution model bakeoff: read these measurements
Read the complete rubric, sample and billing notes
23-model campaign; 69 frozen decisions × 3 samples; 12 published model rows
Choose one next action in an unfinished trading-agent workflow. This replay measures individual decisions, not a fresh complete trading run.
Scoring
Mean action score comes from the judge. Schema validity is the share of actions matching the required output structure; a valid structure can still contain the wrong action. Switching judges reduced the Luna-versus-DeepSeek score gap from 13.9 to 1.5 points.
Cost units
Cost uses historical provider billing per 1,000 decisions. Reused prefixes earned cache discounts; the cached-token split was not retained.
Time units
Prod p50 is median production response latency, separate from replay quality and complete agent wall time.
Five models completing one sandbox research task
On the ship-traffic research task, Gemini 3.5 Flash Lite returned the grade sequence D, then failed out over 24m, with reported run cost —. Read the whole sequence: a favorable review followed by a failure exposes a repair that did not hold. A missing cost remains missing, and uneven CPU throttling limits timing comparisons.
Show the table · 3 rows, 2 columns
| Published metric | Value |
|---|---|
| Grade trajectory | D, then failed out |
| Wall | 24m |
| Cost | — |
No matching rows. Clear the filter to see all records.
Write research code and repair its artifacts: follow the published task
The grade sequence shows whether a repair held up at the next review. One B+ followed by D is a different outcome from repeated favorable reviews.
Run one concrete research task
Correlate Strait of Hormuz ship traffic with the next USO trading session.
Write and run research code on real Modal machines, return the artifacts for review and repair defects raised by the grader.
Inspect the recorded repairs and regressions
Luna returned D → B+ → D → B+ → D → B+; GLM returned D → D → B+ → D → D → D. DeepSeek returned D on all six reviews.
The source found self-check code that assigned the same variable to delivered_stats and recomputed, then hard-coded a match. That explains why a passing self-check could survive while the delivered analysis still failed review. The publication does not assign this particular code defect to one model in the table.
Five models completing one sandbox research task: read these measurements
Read the complete rubric, sample and billing notes
5 models on the same single research task; real Modal virtual machines
Correlate Strait of Hormuz ship traffic with the next USO trading session.
Scoring
Grades describe the delivered artifact after successive reviews. Oscillation exposed holes in agent-written self-checks; a successful process exit was insufficient.
Cost units
Cost is historical billed run cost. Flash Lite has no reported total after failing out; a dash is missing data, not zero.
Time units
Wall time covers the complete workflow. Uneven web-tier CPU throttling affected timing, so the source treats it as a soft comparison.
Sandbox
An isolated machine where the agent runs research code.
Models answering 22 stock questions with SQL
On 22 stock questions, Gemini 3.5 Flash Lite recorded a blended score of 0.664, success rate 72.7% and average workflow time 19.4s. The blended score includes zeros for non-answers; it is different from correctness among answers that reached grading. The example below explains the data task without attributing its answer to this model.
Show the table · 4 rows, 2 columns
| Published metric | Value |
|---|---|
| Average score | 0.664 |
| Median | 0.90 |
| Success rate | 72.7% |
| Avg execution | 19.4s |
No matching rows. Clear the filter to see all records.
Answer a stock question with a data query: follow the published task
This published example explains the task. It is not an identified scored fixture or an answer attributed to any one model. Stored P/E and price divided by the displayed EPS are different fields in its query.
Published worked example, without model attribution
What were the trailing P/E ratios for TSLA and GOOGL on 2025-05-21?
Translate the question into DuckDB SQL, a query over the financial data, then inspect the returned rows and explain the answer.
Read what came back
The original example reported GOOGL at $168.56 with stored P/E 18.46, and TSLA at $334.62 with stored P/E 169.17. Both rows show 2025-05-21.
Inspect both tickers, the requested date and four distinct fiscal quarters of earnings. The original EPS query did not deduplicate fiscal periods, and the correction has not been rerun.
Models answering 22 stock questions with SQL: read these measurements
Read the complete rubric, sample and billing notes
22 natural-language questions; 6 published model rows
Answer financial questions by writing and executing DuckDB SQL against the stock data lake.
Scoring
The average is a blended score across all question cells, including zeros for clarification or execution failure. Success rate and graded-only correctness answer different questions; none measures investment returns.
Cost units
The published screening table has no per-model cost column.
Time units
Average execution time covers the SQL-answer workflow, not the executor study's production p50.
Graded-only mean
The mean among answers that reached grading. Luna recorded 0.931 across 13 graded cells; nine ungraded cells lowered its blended score to 0.550.
TTM earnings / P/E
Trailing twelve-month earnings cover four fiscal quarters. Price-to-earnings relates price to earnings; the published query reads a stored P/E field rather than recomputing it from the displayed EPS.
No comments yet.