Public page collection

Benchmark studies: published pages

Browse the published Benchmark studies collection with direct links and numbered pages.

Published pages: 28
Directory page: 1 of 1

Search this collection

Sort this collection

Pages 1–28

Claude 3.7 Sonnet and o3-mini building a vague leveraged-ETF strategy

Published 2025-02-24: 1 ambiguous strategy request, 2 models, one generated strategy each, backtested once. Backtest statistics of each generated strategy: percent change, Sharpe ratio, Sortino ratio, max and average drawdown, number of trades. See the results, how they were measured and where the findings stop.

Artifact grading and repeatability study

See ten filing transactions become nineteen extracted rows, then follow how independent reviewers reduce their findings. Compare the actual published recall and consistency results.

Next-action execution model bakeoff

Published 2026-08-08: 23-model campaign; 69 frozen decisions × 3 samples; 12 published model rows. Mean next-action score; schema validity, billed cost and production p50 are separate. See the results, how they were measured and where the findings stop.

Gemini 2.0 Flash, DeepSeek R1 and o3-mini writing financial SQL

Published 2025-02-05: 2 natural-language financial questions, 3 models, one attempt each. Author-assigned score per question (0 to 1) and published OpenRouter list price per million tokens. See the results, how they were measured and where the findings stop.

DeepSeek R1 and OpenAI o1 answering SPY drawdown questions with SQL

Published 2025-01-21: 3 chained SPY research questions in one conversation per model, 2 models, single run each. Answer values each model returned per question, the article's correctness verdict on question 1, and list price per million tokens. See the results, how they were measured and where the findings stop.

Best AI voice model in 2026: HeyGen Voice 1 Professional

Published 2026-10-16: Two HeyGen Voice 1 voice clones created on 2026-10-14: a Professional Clone trained on 72m32s of speech and an Instant Clone from one 2m50s recording, each with one checked demo. Which clone the author approved and now uses by default, with each demo's duration and format. See the results, how they were measured and where the findings stop.

Jev 1.14 vs GPT-5.6 Luna: mapper and reducer benchmark

We reviewed saved Jev 1.14 and GPT-5.6 Luna responses separately as mappers and reducers. See defect coverage, final verdicts and a replay of false flags.

Nemotron 3.5 Super VL benchmark: 298 trading-assistant attempts

298 attempts, four full suites, and saved incumbents for comparison. See the grades, recorded costs and response times.

Claude 3.5 Sonnet and GPT-4o turning a screen into a MongoDB query

Published 2024-06-25: 1 screening request, 2 models, single run each. Whether the model returned an answer, and the time to that answer. See the results, how they were measured and where the findings stop.

Eleven models building an options strategy

Published 2026-04-06: 11 model runs; same task, account context, tools and 25-iteration budget. Goal-oriented evaluator score / 100. See the results, how they were measured and where the findings stop.

Planning models on 29 frozen conversations

Published 2026-08-08: 16-model campaign, 29 frozen conversations; 14 model rows shown in the public table. Mean graded plan score; standard error reported separately. See the results, how they were measured and where the findings stop.

Claude 3.5 Sonnet and GPT-4o generating trading portfolio configs

Published 2024-06-25: 2 portfolio-creation prompts, 2 models, single run each. Author-graded correctness of the generated portfolio configuration, approximate time to the final response, and list price per million tokens. See the results, how they were measured and where the findings stop.

GPT-4o mini and GPT-3.5 Turbo generating a portfolio JSON object

Published 2024-07-19: 1 portfolio JSON task, 2 models, prompt count and run count not published in this excerpt. Pass or fail: whether the model produced a valid portfolio JSON object that worked downstream. See the results, how they were measured and where the findings stop.

Five models completing one sandbox research task

Compare every reported grade for five models working on one real sandbox research task. See the regressions, recorded costs and the self-check mistake that a successful run can hide.

ScaleDown Extract Review: Accuracy, Speed and Cost Tested

ScaleDown Extract got GAAP EPS right on 86 of 138 real 8-K earnings releases, against 121 for DeepSeek V4 Flash. It was 12x faster and about 10x cheaper.

ScaleDown Summarize Review: Accuracy, Speed and Cost Tested

ScaleDown Summarize was accurate on 11 of 12 real documents and 15x faster than Muse Spark 1.3, but misread 6 of 13 stock screener tables.

GPT-4o mini and GPT-3.5 Turbo generating a valid SQL query

Published 2024-07-19: 1 SQL generation task, 2 models, prompt count and run count not published in this excerpt. Pass or fail: whether the model produced a syntactically valid query that executed and returned results the author judged correct. See the results, how they were measured and where the findings stop.

DeepSeek R1 and o3-mini writing a dividend stock screen in SQL

Published 2025-02-01: 1 natural-language screening request, 2 models, R1 asked twice, o3-mini once. Author-graded accuracy (0 or 1), estimated cost in units of one R1 request (c), and wall-clock latency. See the results, how they were measured and where the findings stop.

Models generating SQL for financial questions, April 2025

Published 2025-04-15: 60 financial questions per the article text, 11 models. Average, median and standard deviation of query score. Success rate. Average execution time. List input and output price. See the results, how they were measured and where the findings stop.

Models generating financial SQL queries, July 2025

Published 2025-07-11: About 88 financial questions per the article text, 11 models. Median and average judge score (0 to 1), success rate, average response time and list token prices. See the results, how they were measured and where the findings stop.

GPT-5 family and peers generating financial SQL queries

Published 2025-08-10: EvaluateGPT financial question set (count not restated in this article), 8 models. Median and average judge score (0 to 1), success rate, average response time and list token prices. See the results, how they were measured and where the findings stop.

GPT-oss and Gemini generating financial SQL queries

Published 2025-08-06: 90 financial questions per the article text, 4 models. Median and average judge score (0 to 1), success rate, average response time, list token prices and context window. See the results, how they were measured and where the findings stop.

Seven models generating financial SQL queries

Published 2025-04-10: 7 models on a shared SQL question set, the article does not state the number of questions. Per-query score from 0 to 1 summarized as average, median and standard deviation, plus success rate, average execution time in milliseconds and share of perfect scores. See the results, how they were measured and where the findings stop.

Models answering 22 stock questions with SQL

Published 2026-08-08: 22 natural-language questions; 6 published model rows. Average answer score, success rate and average execution time. See the results, how they were measured and where the findings stop.

MongoDB time-series collections vs TimescaleDB for backtest data

Published 2025-10-03: 2 databases, backtest and portfolio-history queries, 20 requests in batches of four per the article. Total storage, average backtest and portfolio query latency, combined latency, and monthly cost. See the results, how they were measured and where the findings stop.

Models turning plain-English trading rules into strategy configs

Published 2025-08-11: 14 models on one sample of strategy requests, this article does not state the count (the companion article of 2025-08-12 describes the same table as 12 financial questions). Median and average semantic judge score (0 to 1), success rate, average generation time in milliseconds, and list input/output price per million tokens. See the results, how they were measured and where the findings stop.

Claude 3.5 Sonnet and o3-mini building a Bitcoin trading strategy

Published 2025-02-07: 1 strategy request, 2 models, one attempt each. Whether the generated buy rule matched the requested condition, judged by the author. See the results, how they were measured and where the findings stop.

Voice-cloning systems ranked blind on one speaker's voice

Published 2026-08-24: 16 ranked arms (fine-tuned and zero-shot variants of what the article calls eleven systems), each reading the same 67-word script, plus a recorded human control. Blind rank by perceived similarity to the author's voice, with the author's verbatim verdict and any reported clip duration or word coverage. See the results, how they were measured and where the findings stop.

Continue browsing

All public collections ↗