TL;DR
Tencent Hy3 found 83.1% of labeled defects and repeated its grade on 18 of 23 artifacts in the August 2026 review study. Luna matched that defect recall and repeated its grade on 19 artifacts. The difference is one repeat agreement in this sample, not a broad review-quality advantage. Reducers judge the findings their reviewers supply; repeated grades alone do not prove correctness. The table publishes no per-model cost or latency.
- Exact tested ID
- tencent/hy3
- Published studies
- 1
- Provider
- tencent
Artifact grading and repeatability study: compare the tested outputs
Both models found 83.1% of labeled defects. The difference is one repeat agreement in this sample; repeated grades alone do not establish correctness.
Show the table · 2 rows, 3 columns
Scroll sideways to compare all columns. Select a heading to sort.
| tencent/hy3 | 83.1% | 18 of 23 |
| openai/gpt-5.6-luna | 83.1% | 19 of 23 |
No matching rows. Clear the filter to see all records.
23 hand-labeled artifacts covering 14 task types; 3 reducer models in the public table. Published 2026-08-08. Values retain their original units and grading scale.
tencent/hy3 was tested on 2026-08-08. 23 hand-labeled artifacts covering 14 task types; 3 reducer models in the public table. Its row and the other published arm use the same task and measurement units.
What this version helps you choose
Hy3 tied Luna’s defect recall and repeated its grade on one fewer artifact. That narrow consistency difference belongs to this 23-artifact sample.
Review defects in delivered research: follow the published task
The reducer can assess only the defects supplied by its mappers. The repeated-grade count measures consistency separately from finding defects.
Source: 10 transactions and a Disney sale
The published Pelosi research checks 65 disclosure PDFs against 135 transactions read independently by eye. In the cited filing, the source contains 10 transactions; the Disney transaction is a sale.
These are public extraction errors used to explain what a reviewer must check.
Returned rows: 19 transactions and a Disney purchase
A sliding-window regex returned 19 transactions for the filing containing 10. Omitting the transaction-type filter turned the Disney sale into a purchase. Both errors left plausible-looking JSONL rows.
Compare the returned count and buy/sell direction with the original filing. A successful code exit does not establish either match.
Review: nine findings become six upheld defects
The published diagram gives OCR text, a rendered image, the native PDF, and code plus rows to four independent reviewers. Their nine findings reach one reducer: three are dropped as noise, six are upheld, and the displayed grade is B−.
This diagram illustrates the review mechanism. It is not identified as one of the 23 private benchmark artifacts or as a scored response from Luna, Hy3 or MiniMax.
Artifact grading and repeatability study
For artifact review, Hy3 found 83.1% of the hand-labeled defects and repeated its grade on 18 of 23 artifacts. These are separate measurements of finding defects and consistency. Cost and latency are not published for this comparison.
Show the table · 2 rows, 2 columns
| Published metric | Value |
|---|---|
| Defects found | 83.1% |
| Same grade on re-run | 18 of 23 |
No matching rows. Clear the filter to see all records.
Artifact grading and repeatability study: read these measurements
Read the complete rubric, sample and billing notes
23 hand-labeled artifacts covering 14 task types; 3 reducer models in the public table
Judge defects in delivered research artifacts, including code and validation evidence.
Scoring
Defect recall is the share of hand-labeled defects found. Repeated-grade agreement counts artifacts assigned the same grade on a repeat; agreement alone does not prove correctness.
Cost units
This comparison table has no per-model billing column.
Time units
This table publishes no per-model latency.
Mapper / reducer
Mappers inspect parts of an artifact independently. The reducer combines their findings and assigns the final grade.
No comments yet.