TL;DR
Luna found 83.1% of labeled defects and MiniMax M3 found 82.5% in the August 2026 artifact-review comparison. Luna repeated its grade on 19 of 23 artifacts, versus MiniMax's 16. That supports a repeatability preference in this small sample; the narrow recall gap is not a universal review advantage. Both reducers received the same reviewer findings and could not recover omitted defects. The published table has no per-model cost or latency.
Artifact grading and repeatability study: published results
Luna found 83.1% of labeled defects and MiniMax M3 found 82.5%. Repeated-grade counts measure consistency, not proof of correctness. Cost and latency were not published.
Show the table · 2 rows, 3 columns
Scroll sideways to compare all columns. Select a heading to sort.
| openai/gpt-5.6-luna | 83.1% | 19 of 23 |
| minimax/minimax-m3 | 82.5% | 16 of 23 |
No matching rows. Clear the filter to see all records.
23 hand-labeled artifacts covering 14 task types; 3 reducer models in the public table. Published 2026-08-08.
Which fits the tested job?
Luna found slightly more labeled defects and repeated its grade more often than MiniMax M3 in this small artifact sample. That supports testing Luna when repeatability matters, while the narrow recall gap does not establish a universal review advantage.
Artifact grading and repeatability study
Reported defect recall was 83.1% for Luna and 82.5% for MiniMax; repeated grades matched in 19 of 23 and 16 of 23 reruns. Each reducer received the same mapper findings and could not recover a defect absent from them. No per-model cost or latency column is published, so this comparison cannot price that repeatability tradeoff.
Review defects in delivered research: follow the published task
The reducer can assess only the defects supplied by its mappers. The repeated-grade count measures consistency separately from finding defects.
Source: 10 transactions and a Disney sale
The published Pelosi research checks 65 disclosure PDFs against 135 transactions read independently by eye. In the cited filing, the source contains 10 transactions; the Disney transaction is a sale.
These are public extraction errors used to explain what a reviewer must check.
Returned rows: 19 transactions and a Disney purchase
A sliding-window regex returned 19 transactions for the filing containing 10. Omitting the transaction-type filter turned the Disney sale into a purchase. Both errors left plausible-looking JSONL rows.
Compare the returned count and buy/sell direction with the original filing. A successful code exit does not establish either match.
Review: nine findings become six upheld defects
The published diagram gives OCR text, a rendered image, the native PDF, and code plus rows to four independent reviewers. Their nine findings reach one reducer: three are dropped as noise, six are upheld, and the displayed grade is B−.
This diagram illustrates the review mechanism. It is not identified as one of the 23 private benchmark artifacts or as a scored response from Luna, Hy3 or MiniMax.
Results from tasks both models attempted
The tested model IDs are openai/gpt-5.6-luna and minimax/minimax-m3. Every result table below comes from a study in which both participated.
Compare the measured quality, historical billed cost and elapsed time within the same task and rubric.
Scores from different tasks are not combined into one winner score.
Artifact grading and repeatability study: read these measurements
Read the complete rubric, sample and billing notes
23 hand-labeled artifacts covering 14 task types; 3 reducer models in the public table
Judge defects in delivered research artifacts, including code and validation evidence.
Scoring
Defect recall is the share of hand-labeled defects found. Repeated-grade agreement counts artifacts assigned the same grade on a repeat; agreement alone does not prove correctness.
Cost units
This comparison table has no per-model billing column.
Time units
This table publishes no per-model latency.
Mapper / reducer
Mappers inspect parts of an artifact independently. The reducer combines their findings and assigns the final grade.
No comments yet.