TL;DR
MiniMax M3 found 82.5% of labeled defects and repeated its grade on 16 of 23 artifacts in the August 8, 2026 review study. Luna found 83.1% and repeated 19 grades. The sample covered 14 task types, with no published per-model cost or latency. Finding a defect and giving a stable grade are separate measurements, and the reducer can assess only the findings its reviewers supply.
- Exact tested ID
- minimax/minimax-m3
- Published studies
- 1
- Provider
- minimax
Artifact grading and repeatability study: compare the tested outputs
August 8, 2026: counts from twenty-three artifacts. Recall remains 82.5% and 83.1% respectively in the unchanged table; agreement and defect detection answer different questions.
Show the table · 2 rows, 3 columns
Scroll sideways to compare all columns. Select a heading to sort.
| minimax/minimax-m3 | 82.5% | 16 of 23 |
| openai/gpt-5.6-luna | 83.1% | 19 of 23 |
No matching rows. Clear the filter to see all records.
23 hand-labeled artifacts covering 14 task types; 3 reducer models in the public table. Published 2026-08-08. Values retain their original units and grading scale.
minimax/minimax-m3 was tested on 2026-08-08. 23 hand-labeled artifacts covering 14 task types; 3 reducer models in the public table. Its row and the other published arm use the same task and measurement units.
What this version helps you choose
MiniMax’s defect recall was close to Luna’s, while it repeated its grade on fewer of the 23 artifacts. Cost and timing are not published.
Review defects in delivered research: follow the published task
The reducer can assess only the defects supplied by its mappers. The repeated-grade count measures consistency separately from finding defects.
Source: 10 transactions and a Disney sale
The published Pelosi research checks 65 disclosure PDFs against 135 transactions read independently by eye. In the cited filing, the source contains 10 transactions; the Disney transaction is a sale.
These are public extraction errors used to explain what a reviewer must check.
Returned rows: 19 transactions and a Disney purchase
A sliding-window regex returned 19 transactions for the filing containing 10. Omitting the transaction-type filter turned the Disney sale into a purchase. Both errors left plausible-looking JSONL rows.
Compare the returned count and buy/sell direction with the original filing. A successful code exit does not establish either match.
Review: nine findings become six upheld defects
The published diagram gives OCR text, a rendered image, the native PDF, and code plus rows to four independent reviewers. Their nine findings reach one reducer: three are dropped as noise, six are upheld, and the displayed grade is B−.
This diagram illustrates the review mechanism. It is not identified as one of the 23 private benchmark artifacts or as a scored response from Luna, Hy3 or MiniMax.
Artifact grading and repeatability study
For artifact review, MiniMax M3 found 82.5% of the hand-labeled defects and repeated its grade on 16 of 23 artifacts. These are separate measurements of finding defects and consistency. Cost and latency are not published for this comparison.
Show the table · 2 rows, 2 columns
| Published metric | Value |
|---|---|
| Defects found | 82.5% |
| Same grade on re-run | 16 of 23 |
No matching rows. Clear the filter to see all records.
Artifact grading and repeatability study: read these measurements
Read the complete rubric, sample and billing notes
23 hand-labeled artifacts covering 14 task types; 3 reducer models in the public table
Judge defects in delivered research artifacts, including code and validation evidence.
Scoring
Defect recall is the share of hand-labeled defects found. Repeated-grade agreement counts artifacts assigned the same grade on a repeat; agreement alone does not prove correctness.
Cost units
This comparison table has no per-model billing column.
Time units
This table publishes no per-model latency.
Mapper / reducer
Mappers inspect parts of an artifact independently. The reducer combines their findings and assigns the final grade.
No comments yet.