TL;DR
Luna was the strongest of five models on the published August 8, 2026 ship-traffic research task, reaching B+ three times at a recorded $2.63. Its intervening D grades show that repairs did not consistently hold. GLM reached B+ once before regressing, while DeepSeek received six D grades. This single task supports a workload-specific choice; uneven CPU throttling limits speed comparisons, and self-checks must independently compare delivered artifacts with sources.
Follow the task and all reported repairs
See every result for the same research task
The task was to correlate Strait of Hormuz ship traffic with the next USO trading session. Each model worked on real Modal machines with the same planner, grader and remediation loop. Luna reached B+ three times, with a D grade after each of the first two. GLM reached B+ once before returning to D.
The one-task outcome favors Luna for this repair loop. Wall-clock time and billing retain their original units; varied time limits and CPU throttling limit a speed comparison.
Check whether the self-check is independent
The publication found code that assigned the same variable to delivered_stats and recomputed, then hard-coded a match. Comparing a variable with itself cannot check whether the delivered analysis matches the source.
Inspect where each checked value comes from and what the assertion actually compares. This defect is described in the publication but is not attributed to one model in the table.
Read the repair sequence in Aurora
Open the prepared request in Aurora. It includes the example’s actual inputs, so you can ask about the part you want to check.
Review the explanation against the evidence on this page. Account changes and research jobs require a separate decision.
What the result helps you choose
Read what the result helps you choose
Luna's sequence was D → B+ → D → B+ → D → B+, over 59 minutes at a recorded $2.63. GLM reached B+ once and then regressed to D. The one-task result favors Luna for this repair loop while retaining the failed checks and timing limits.
Task and tested input
Read task and tested input
Correlate Strait of Hormuz ship traffic with the next USO trading session.
Instruction in brief (task summary)
Write and run the research code, inspect its artifacts, and fix defects raised by review. This is the published task description, not the full private prompt.
Published results
Not plotted because no numeric value was recorded: google/gemini-3.5-flash-lite.
5 models on the same single research task; real Modal virtual machines. Published 2026-08-08. Values retain their original units and grading scale.
Show the table · 5 rows, 4 columns
Scroll sideways to compare all columns. Select a heading to sort.
| openai/gpt-5.6-luna | D → B+ → D → B+ → D → B+ | 59m | $2.63 |
| z-ai/glm-5.2 | D → D → B+ → D → D → D | 91m | $2.72 |
| qwen/qwen3.7-flash | D → D → D | 124m | $2.85 |
| deepseek/deepseek-v4-flash | D → D → D → D → D → D | 181m | $5.17 |
| google/gemini-3.5-flash-lite | D, then failed out | 24m | — |
No matching rows. Clear the filter to see all records.
5 models on the same single research task; real Modal virtual machines. Published 2026-08-08. Values retain their original units and grading scale.
Calculate the decision from the measured rows
Show the table · 2 rows, 4 columns
Scroll sideways to compare all columns. Select a heading to sort.
| openai/gpt-5.6-luna | D → B+ → D → B+ → D → B+ | 59m | $2.63 |
| deepseek/deepseek-v4-flash | D → D → D → D → D → D | 181m | $5.17 |
No matching rows. Clear the filter to see all records.
Reported whole-run costs were $2.63 and $5.17; the difference is $2.54. Their recorded wall times were 59m and 181m. Read the full grade trajectories: Luna reached B+ repeatedly while DeepSeek's published trajectory stayed at D.
Use the artifact's independent checks as the acceptance criterion, not a successful code exit or its final grade alone. CPU throttling and the single-task sample limit a general speed comparison; missing cost on the failed Flash Lite run remains missing.
How the runs worked
Read the test procedure
5 models on the same single research task; real Modal virtual machines
NexusTrade sandbox operator with the same planner, grader and remediation loop
Step 1
Keep the same task, planner and grader for five operator models.
Step 2
Run code on real Modal machines and return artifacts for review.
Step 3
Feed findings into remediation and retain the whole grade trajectory, including regressions.
How to read the measurements
Read how to read the measurements
Grading and valid output
Grades describe the delivered artifact after successive reviews. Oscillation exposed holes in agent-written self-checks; a successful process exit was insufficient.
Cost
Cost is historical billed run cost. Flash Lite has no reported total after failing out; a dash is missing data, not zero.
Time
Wall time covers the complete workflow. Uneven web-tier CPU throttling affected timing, so the source treats it as a soft comparison.
Sandbox
An isolated machine where the agent runs research code.
Sample and scope
Read sample and scope
This is one task, not a broad coding benchmark. A favorable result cannot be generalized to all trading research code.
The early remediation loop oscillated between grades. Later diagnoses found flaws in agent-written audit code; a successful execution status alone did not establish the correctness of delivered analysis.
Flash Lite failed out after 24 minutes and has no reported total cost in the table. Missing cost is not represented as zero.
Read the original evidence
Aurora product context
See the broader product workflow
This August recording shows the broader Aurora agent workflow. The task explanations above describe each evaluation; its private conversations and complete system prompts remain private.
No comments yet.