TL;DR
Choose a model for the task being measured. The April 2026 options study’s leading Flash model scored 66/100, but the evaluator called its result mixed and asked for more iteration. August studies favored GPT 5.6 Luna for cost-conscious planning and execution, while Gemini 3.6 Flash led stock SQL screening. These studies measure different workflows and rubrics; neither a high evaluator score nor a deployment action establishes verified live trading returns.
- Published task studies
- 6
- Latest publication
- 2026-08-08
- Live-return leaderboard
- Not measured
Aurora product context
See the broader product workflow
This August recording shows the broader Aurora agent workflow. The task explanations above describe each evaluation; its private conversations and complete system prompts remain private.
Choose the model for the job
The published evidence does not identify one universal best trading model. The options-workflow study favored a Google Flash model on its rubric; the later planning and execution studies favored a cost-efficient OpenAI version for different reasons; SQL screening favored a different Google version.
A model supplies reasoning inside a harness. Data access, executable strategy rules, simulation infrastructure, schema validation, iteration limits and review determine whether it can actually complete a trading task. Meta Muse and Grok Bot are agent products; Muse Spark and Grok model versions are evaluated separately here.
Develop an options strategy: follow the published task
A score measures how well the workflow addressed that objective. Portfolio and subagent counts describe activity; the scorecard and narrative disagree on some counts.
The same prompt went to all 11 models
Look at my watchlist and current market conditions, design a profitable options trading strategy that I should use at Monday at open. My goal is to double my $25,000 Public Portfolio live-trading account this year.
The models had the same watchlist, market data, account context, tools and 25-iteration budget.
Flash acted; GPT-5.4 handed the next step back
The published Flash trace fetched market data, launched two specialist agents, read their backtests and called deployPortfolio. GPT-5.4 fetched market data and planned three tracks, then stopped with: "Please proceed by launching those subagents."
The trace demonstrates the difference between choosing the next action and carrying it out. Its Flash process counts disagree with the preserved scorecard.
Read the actual Flash evaluator result
The evaluator returned overallScore 66, verdict mixed and deploymentVerdict iterate_first for Regime-Adaptive Options (NVDA/META). Its listed yearly returns were 2022 +31.22%, 2023 +15.41% and 2024 +0.34%, averaging +15.66%.
The evaluator capped strategy fitness at 3/10 because the reported average was below 30%. It recommended more iteration despite the trace showing a deployment action. Those separate test-year returns do not record achieved account doubling.
Published findings by task
Show the table · 6 rows, 3 columns
Scroll sideways to compare all columns. Select a heading to sort.
| Options Strategy Development | 2026-04-06 | Gemini 3 Flash Preview has the highest published score, 66/100. The evaluator still calls that mixed and asks for more iteration; no result proves the account-doubling objective. |
| Agent Planning | 2026-08-08 | GPT 5.6 Luna was the published cost-oriented selection among similar-quality planning candidates. Muse Spark 1.1 had the highest raw mean; those are different claims. |
| Agent Execution | 2026-08-08 | The published deployment decision selected GPT 5.6 Luna because it improved score, schema validity, billed cost and latency versus the old DeepSeek default. GPT 5.6 Luna Pro had a higher raw score and a substantially higher cost. |
| Artifact Review | 2026-08-08 | Luna and Hy3 share the reported recall. Luna repeated its grade on 19/23 reruns, Hy3 on 18/23 and MiniMax M3 on 16/23. |
| Sandbox Code Generation | 2026-08-08 | Luna reached B+ three times and had the lowest reported completed-run wall time and cost. The oscillating grades and single-task sample remain material limitations. |
| Stock Screening | 2026-08-08 | Gemini 3.6 Flash has the highest published average answer score (0.841) and success rate (86.4%) in this six-row SQL comparison. GPT 5.6 Luna scores 0.550 on the same table. |
No matching rows. Clear the filter to see all records.
What each model was asked to do
Options Strategy Development
Develop an options strategy for a $25,000 account with the stated goal of doubling it within a year.
Agent Planning
Route an incoming request: write a plan, call a catalog tool, ask for clarification or continue an existing agent.
Agent Execution
Choose one next action in an unfinished trading-agent workflow. This replay measures individual decisions, not a fresh complete trading run.
Artifact Review
Judge defects in delivered research artifacts, including code and validation evidence.
Sandbox Code Generation
Correlate Strait of Hormuz ship traffic with the next USO trading session.
Stock Screening
Answer financial questions by writing and executing DuckDB SQL against the stock data lake.
How to compare the evidence
Same task and harness
Compare models on the same inputs, tools and output contract. Record the exact version IDs.
More than one favorable result
Report repeated samples, uncertainty and failure behavior when available. A one-task run is a screen, not a universal verdict.
Separate measurements
Quality, schema validity, elapsed time, provider-billed cost and trading returns have different units.
Inspect the judge
The published execution regrade showed vendor-dependent score gaps. Calibration and deterministic checks matter.
Historical evidence and current decisions
The results below retain their publication dates and original units. August deployment decisions are historical observations, not a claim about current defaults. Prices and model availability may have changed.
Evaluator confidence, historical backtest returns, a deployment action and verified live account performance measure different things. Evidence for one does not establish the others.
No comments yet.