TL;DR
Qwen3.7 Flash's three artifact reviews all remained at D on the ship-traffic and USO research task published August 8, 2026. Its workflow took 124 minutes and cost $2.85. The five-model comparison ran research code on real sandbox machines and reviewed repairs, rather than only grading a plan. Uneven CPU throttling makes wall time a softer comparison, while the repeated failed reviews remain visible.
- Exact tested ID
- qwen/qwen3.7-flash
- Published studies
- 1
- Provider
- qwen
Five models completing one sandbox research task: compare the tested outputs
August 8, 2026: one research task on actual sandbox machines. Uneven CPU throttling limits timing comparison; Qwen’s D to D to D and Luna’s oscillating grades remain in the table.
Show the table · 2 rows, 4 columns
Scroll sideways to compare all columns. Select a heading to sort.
| qwen/qwen3.7-flash | D → D → D | 124m | $2.85 |
| openai/gpt-5.6-luna | D → B+ → D → B+ → D → B+ | 59m | $2.63 |
No matching rows. Clear the filter to see all records.
5 models on the same single research task; real Modal virtual machines. Published 2026-08-08. Values retain their original units and grading scale.
qwen/qwen3.7-flash was tested on 2026-08-08. 5 models on the same single research task; real Modal virtual machines. Its row and the other published arm use the same task and measurement units.
What this version helps you choose
Its three sandbox artifact grades stayed at D. The 124-minute workflow did not reach the favorable grade Luna reached on the same task.
Write research code and repair its artifacts: follow the published task
The grade sequence shows whether a repair held up at the next review. One B+ followed by D is a different outcome from repeated favorable reviews.
Run one concrete research task
Correlate Strait of Hormuz ship traffic with the next USO trading session.
Write and run research code on real Modal machines, return the artifacts for review and repair defects raised by the grader.
Inspect the recorded repairs and regressions
Luna returned D → B+ → D → B+ → D → B+; GLM returned D → D → B+ → D → D → D. DeepSeek returned D on all six reviews.
The source found self-check code that assigned the same variable to delivered_stats and recomputed, then hard-coded a match. That explains why a passing self-check could survive while the delivered analysis still failed review. The publication does not assign this particular code defect to one model in the table.
Five models completing one sandbox research task
On the ship-traffic research task, Qwen3.7 Flash returned the grade sequence D → D → D over 124m, with reported run cost $2.85. Read the whole sequence: a favorable review followed by a failure exposes a repair that did not hold. A missing cost remains missing, and uneven CPU throttling limits timing comparisons.
Show the table · 3 rows, 2 columns
| Published metric | Value |
|---|---|
| Grade trajectory | D → D → D |
| Wall | 124m |
| Cost | $2.85 |
No matching rows. Clear the filter to see all records.
Five models completing one sandbox research task: read these measurements
Read the complete rubric, sample and billing notes
5 models on the same single research task; real Modal virtual machines
Correlate Strait of Hormuz ship traffic with the next USO trading session.
Scoring
Grades describe the delivered artifact after successive reviews. Oscillation exposed holes in agent-written self-checks; a successful process exit was insufficient.
Cost units
Cost is historical billed run cost. Flash Lite has no reported total after failing out; a dash is missing data, not zero.
Time units
Wall time covers the complete workflow. Uneven web-tier CPU throttling affected timing, so the source treats it as a soft comparison.
Sandbox
An isolated machine where the agent runs research code.
No comments yet.