TL;DR
Luna and Hy3 each found 83.1% of labeled defects in the August 2026 artifact review study. Luna repeated its grade on 19 of 23 artifacts, versus Hy3's 18 and MiniMax's 16. That supports a small consistency preference in this sample, with tied recall. A separate public filing example shows why successful extraction can still contain duplicate rows and reversed trades. Reducer quality remains limited by the findings its reviewers supply.
Follow an error through the review
Compare the rows with the source
The published Pelosi research checked 65 disclosure PDFs against 135 transactions read independently by eye. In the cited filing, a sliding-window regex returned 19 transactions instead of 10 and labeled a Disney sale as a purchase.
Count the returned rows and check buy/sell direction against the filing. Plausible JSONL and a successful exit do not establish either match.
Reduce the findings before assigning a grade
The published review diagram sends four views of the artifact to independent reviewers. Nine findings reach the reducer; it drops three as noise, upholds six and displays B-minus.
This is the published mechanism illustration. It is separate from the private benchmark set and does not identify a scored response from Luna, Hy3 or MiniMax.
Compare finding defects with repeating a grade
Luna and Hy3 tied at 83.1% reported defect recall. Their grades repeated on 19 and 18 of 23 artifacts respectively. MiniMax found 82.5% of defects and repeated its grade on 16 of 23.
Luna has one more repeated grade than Hy3 in this sample. The reducer still depends on the defects its mappers supply, and these results provide no model costs or latencies.
Inspect the extraction errors in Aurora
Open the prepared request in Aurora. It includes the example’s actual inputs, so you can ask about the part you want to check.
Review the explanation against the evidence on this page. Account changes and research jobs require a separate decision.
What the result helps you choose
Read what the result helps you choose
Luna and Hy3 tied at 83.1% defect recall. Luna repeated its grade on 19 of 23 artifacts, Hy3 on 18 and MiniMax on 16. That one-artifact difference supports a consistency preference in this sample; the table publishes no model cost or latency.
Task and tested input
Read task and tested input
Judge defects in delivered research artifacts, including code and validation evidence.
Instruction in brief (task summary)
Given mapper findings, decide which defects are supported and assign a grade. A public example checks 65 Pelosi disclosure PDFs against 135 transactions read independently by eye.
Published results
23 hand-labeled artifacts covering 14 task types; 3 reducer models in the public table. Published 2026-08-08. Values retain their original units and grading scale.
Show the table · 3 rows, 3 columns
Scroll sideways to compare all columns. Select a heading to sort.
| openai/gpt-5.6-luna | 83.1% | 19 of 23 |
| tencent/hy3 | 83.1% | 18 of 23 |
| minimax/minimax-m3 | 82.5% | 16 of 23 |
No matching rows. Clear the filter to see all records.
23 hand-labeled artifacts covering 14 task types; 3 reducer models in the public table. Published 2026-08-08. Values retain their original units and grading scale.
Calculate the decision from the measured rows
Show the table · 2 rows, 3 columns
Scroll sideways to compare all columns. Select a heading to sort.
| openai/gpt-5.6-luna | 83.1% | 19 of 23 |
| tencent/hy3 | 83.1% | 18 of 23 |
No matching rows. Clear the filter to see all records.
Both reducers found 83.1% of the hand-labeled defects. Luna repeated its grade on 19/23 reruns (82.61%), compared with Hy3's 18/23 (78.26%): one additional agreement in 23 tests. The measurable difference is consistency in this sample; recall is tied.
The reducers receive the same mapper findings. A missed mapper finding cannot be recovered by choosing a stronger reducer. Inspect the actual duplicate-row case on this page before interpreting the aggregate score as end-to-end review coverage.
How the runs worked
Read the test procedure
23 hand-labeled artifacts covering 14 task types; 3 reducer models in the public table
Map/reduce artifact grader versus independently hand-labeled defects
Step 1
Label defects in 23 artifacts across 14 task types.
Step 2
Give each reducer exactly the same mapper findings.
Step 3
Compare defect recall and whether rerunning keeps the same grade.
How to read the measurements
Read how to read the measurements
Grading and valid output
Defect recall is the share of hand-labeled defects found. Repeated-grade agreement counts artifacts assigned the same grade on a repeat; agreement alone does not prove correctness.
Cost
This comparison table has no per-model billing column.
Time
This table publishes no per-model latency.
Mapper / reducer
Mappers inspect parts of an artifact independently. The reducer combines their findings and assigns the final grade.
Sample and scope
Read sample and scope
A reducer cannot discover a defect absent from its mapper inputs. These results do not establish end-to-end recall for every arbitrary artifact.
GPT 5.6 Luna and Tencent Hy3 tie on reported defect recall (83.1%). Their repeated-grade counts differ by one in 23 tests; that is not evidence of a universal review advantage.
This is a task-specific snapshot. Current grader defaults and newer model versions are not inferred from an August publication.
Read the original evidence
Read read the original evidence
Aurora product context
See the broader product workflow
This August recording shows the broader Aurora agent workflow. The task explanations above describe each evaluation; its private conversations and complete system prompts remain private.
No comments yet.