TL;DR
Luna finished the published August 2026 sandbox research task with repeated B+ reviews after failures, at a recorded $2.63 over 59 minutes. GLM reached B+ once then regressed, while DeepSeek received D across six reviews. That favors Luna for this one repair loop, not all coding tasks. Inspect the whole grade trajectory and independently check delivered artifacts: later diagnoses found flawed self checks. Uneven CPU throttling limits speed comparisons, and the failed Flash Lite run has no reported total cost.
Published results
Not plotted because no numeric value was recorded: google/gemini-3.5-flash-lite.
5 models on the same single research task; real Modal virtual machines. Published 2026-08-08. Values retain their original units and grading scale.
Show the table · 5 rows, 4 columns
Scroll sideways to compare all columns. Select a heading to sort.
| openai/gpt-5.6-luna | D → B+ → D → B+ → D → B+ | 59m | $2.63 |
| z-ai/glm-5.2 | D → D → B+ → D → D → D | 91m | $2.72 |
| qwen/qwen3.7-flash | D → D → D | 124m | $2.85 |
| deepseek/deepseek-v4-flash | D → D → D → D → D → D | 181m | $5.17 |
| google/gemini-3.5-flash-lite | D, then failed out | 24m | — |
No matching rows. Clear the filter to see all records.
5 models on the same single research task; real Modal virtual machines. Published 2026-08-08. Values retain their original units and grading scale.
What the result helps you choose
Luna's sequence was D → B+ → D → B+ → D → B+, over 59 minutes at a recorded $2.63. GLM reached B+ once and then regressed to D. The one-task result favors Luna for this repair loop while retaining the failed checks and timing limits.
Write research code and repair its artifacts: follow the published task
The grade sequence shows whether a repair held up at the next review. One B+ followed by D is a different outcome from repeated favorable reviews.
Run one concrete research task
Correlate Strait of Hormuz ship traffic with the next USO trading session.
Write and run research code on real Modal machines, return the artifacts for review and repair defects raised by the grader.
Inspect the recorded repairs and regressions
Luna returned D → B+ → D → B+ → D → B+; GLM returned D → D → B+ → D → D → D. DeepSeek returned D on all six reviews.
The source found self-check code that assigned the same variable to delivered_stats and recomputed, then hard-coded a match. That explains why a passing self-check could survive while the delivered analysis still failed review. The publication does not assign this particular code defect to one model in the table.
Task and tested input
Correlate Strait of Hormuz ship traffic with the next USO trading session.
Instruction in brief (task summary)
Write and run the research code, inspect its artifacts, and fix defects raised by review. This is the published task description, not the full private prompt.
Test against your own workload
Build a follow-up evaluation for your workload
Check whether the tested task and required output match your workload. Use a fixed set of representative examples, repeat the runs and inspect failures. Compare cost and latency alongside quality.
Each study lists the publication date and exact model versions. Use those dates when comparing the historical results with a model you can run today.
Calculate the decision from the measured rows
One August 2026 ship-traffic research task. Luna's six reviews alternated D and B+; DeepSeek's six stayed D. Historical wall times were 59 and 181 minutes, affected by uneven CPU throttling. Flash Lite failed out with no reported total cost.
Show the table · 2 rows, 4 columns
Scroll sideways to compare all columns. Select a heading to sort.
| openai/gpt-5.6-luna | D → B+ → D → B+ → D → B+ | 59m | $2.63 |
| deepseek/deepseek-v4-flash | D → D → D → D → D → D | 181m | $5.17 |
No matching rows. Clear the filter to see all records.
Reported whole-run costs were $2.63 and $5.17; the difference is $2.54. Their recorded wall times were 59m and 181m. Read the full grade trajectories: Luna reached B+ repeatedly while DeepSeek's published trajectory stayed at D.
Use the artifact's independent checks as the acceptance criterion, not a successful code exit or its final grade alone. CPU throttling and the single-task sample limit a general speed comparison; missing cost on the failed Flash Lite run remains missing.
How the runs worked
Read the test procedure
5 models on the same single research task; real Modal virtual machines
NexusTrade sandbox operator with the same planner, grader and remediation loop
Step 1
Keep the same task, planner and grader for five operator models.
Step 2
Run code on real Modal machines and return artifacts for review.
Step 3
Feed findings into remediation and retain the whole grade trajectory, including regressions.
How to read the measurements
Grading and valid output
Grades describe the delivered artifact after successive reviews. Oscillation exposed holes in agent-written self-checks; a successful process exit was insufficient.
Cost
Cost is historical billed run cost. Flash Lite has no reported total after failing out; a dash is missing data, not zero.
Time
Wall time covers the complete workflow. Uneven web-tier CPU throttling affected timing, so the source treats it as a soft comparison.
Sandbox
An isolated machine where the agent runs research code.
Sample and scope
This is one task, not a broad coding benchmark. A favorable result cannot be generalized to all trading research code.
The early remediation loop oscillated between grades. Later diagnoses found flaws in agent-written audit code; a successful execution status alone did not establish the correctness of delivered analysis.
Flash Lite failed out after 24 minutes and has no reported total cost in the table. Missing cost is not represented as zero.
Read the original evidence
Aurora product context
See the broader product workflow
This August recording shows the broader Aurora agent workflow. The task explanations above describe each evaluation; its private conversations and complete system prompts remain private.
No comments yet.