TL;DR
GPT 5 Mini had a 68.2% success rate across 22 stock SQL questions in the August 8, 2026 comparison. Its blended score was 0.664 and average workflow time was 41.5 seconds, compared with Gemini 3.6 Flash's 86.4% success rate and 26.6 seconds. The score includes non-answer zeros, and the worked financial query is not an answer attributed to GPT 5 Mini.
Two earlier studies also include GPT 5 Mini. In the August 10, 2025 financial SQL comparison it recorded a 0.933 median and 0.717 average score with a 78.65% success rate. In the August 11, 2025 strategy-configuration comparison it recorded a 0.933 median and 0.717 average score with a 90.9% success rate. Each result belongs to its own task and grading method.
- Exact tested ID
- openai/gpt-5-mini
- Published studies
- 3
- Provider
- openai
Models answering 22 stock questions with SQL: compare the tested outputs
These exact August 8, 2026 table cells describe the 22-question SQL task. Labels are rounded; the full table retains blended scores and workflow times.
Show the table · 2 rows, 5 columns
Scroll sideways to compare all columns. Select a heading to sort.
| openai/gpt-5-mini | 0.664 | 1.00 | 68.2% | 41.5s |
| google/gemini-3.6-flash | 0.841 | 1.00 | 86.4% | 26.6s |
No matching rows. Clear the filter to see all records.
Original units preserved. Average score is the historical blended measure; this table is not a new run.
openai/gpt-5-mini was tested on 2026-08-08. 22 natural-language questions; 6 published model rows. Its row and the other published arm use the same task and measurement units.
What this version helps you choose
Its SQL blended score tied Flash Lite’s, with a lower success rate and longer average workflow time.
Answer a stock question with a data query: follow the published task
This published example explains the task. It is not an identified scored fixture or an answer attributed to any one model. Stored P/E and price divided by the displayed EPS are different fields in its query.
Published worked example, without model attribution
What were the trailing P/E ratios for TSLA and GOOGL on 2025-05-21?
Translate the question into DuckDB SQL, a query over the financial data, then inspect the returned rows and explain the answer.
Read what came back
The original example reported GOOGL at $168.56 with stored P/E 18.46, and TSLA at $334.62 with stored P/E 169.17. Both rows show 2025-05-21.
Inspect both tickers, the requested date and four distinct fiscal quarters of earnings. The original EPS query did not deduplicate fiscal periods, and the correction has not been rerun.
Models answering 22 stock questions with SQL
Inspect the original GPT 5 Mini metrics
On 22 stock questions, GPT 5 Mini recorded a blended score of 0.664, success rate 68.2% and average workflow time 41.5s. The blended score includes zeros for non-answers; it is different from correctness among answers that reached grading. The example below explains the data task without attributing its answer to this model.
| Published metric | Value |
|---|---|
| Average score | 0.664 |
| Median | 1.00 |
| Success rate | 68.2% |
| Avg execution | 41.5s |
No matching rows. Clear the filter to see all records.
Models answering 22 stock questions with SQL: read these measurements
Read the complete rubric, sample and billing notes
22 natural-language questions; 6 published model rows
Answer financial questions by writing and executing DuckDB SQL against the stock data lake.
Scoring
The average is a blended score across all question cells, including zeros for clarification or execution failure. Success rate and graded-only correctness answer different questions; none measures investment returns.
Cost units
The published screening table has no per-model cost column.
Time units
Average execution time covers the SQL-answer workflow, not the executor study's production p50.
Graded-only mean
The mean among answers that reached grading. Luna recorded 0.931 across 13 graded cells; nine ungraded cells lowered its blended score to 0.550.
TTM earnings / P/E
Trailing twelve-month earnings cover four fiscal quarters. Price-to-earnings relates price to earnings; the published query reads a stored P/E field rather than recomputing it from the displayed EPS.
GPT-5 family and peers generating financial SQL queries
Open every published row
GPT 5 Mini recorded 0.933 for median score in the 2025-08-10 publication. EvaluateGPT financial question set (count not restated in this article), 8 models.
| Published metric | Value |
|---|---|
| Median score | 0.933 |
| Avg score | 0.717 |
| Success rate | 78.65% |
| Avg time | 58,097 ms |
| Input cost | $0.25/M |
| Output cost | $2.00/M |
No matching rows. Clear the filter to see all records.
GPT-5 family and peers generating financial SQL queries: read these measurements
View task, sample and measurement units
EvaluateGPT financial question set (count not restated in this article), 8 models
Generate syntactically and semantically valid SQL queries for real-world financial questions.
Scoring
LLM-judge scores from 0 to 1 as described in the linked GPT-oss article (Claude Sonnet 4, Gemini 2.5 Pro and GPT-4.1 judges in that article). This article does not restate the judges.
Cost units
List prices per million tokens, from $0.05/M input and $0.40/M output (GPT-5 Nano) to $1.25/M input and $10.00/M output (Gemini 2.5 Pro, GPT-5, GPT-5 Chat). Total spend was reported as around $200.
Time units
Average time in milliseconds, from 40,236 ms (Gemini 2.5 Flash) to 113,743 ms (GPT-5).
Models turning plain-English trading rules into strategy configs
Open every published row
GPT 5 Mini recorded 0.933 for median score in the 2025-08-11 publication. 14 models on one sample of strategy requests, this article does not state the count (the companion article of 2025-08-12 describes the same table as 12 financial questions).
| Published metric | Value |
|---|---|
| Median score | 0.933 |
| Avg score | 0.717 |
| Success rate | 90.9% |
| Avg time (ms) | 143,849 |
| Input cost | $0.25/M |
| Output cost | $2.00/M |
No matching rows. Clear the filter to see all records.
Models turning plain-English trading rules into strategy configs: read these measurements
View task, sample and measurement units
14 models on one sample of strategy requests, this article does not state the count (the companion article of 2025-08-12 describes the same table as 12 financial questions)
Convert a natural-language trading strategy request into a NexusTrade portfolio configuration of actions and conditions.
Scoring
LLM judges score each configuration from 0 to 1 using published guidelines (1.0 perfect match, 0.8 to 0.9 minor issues, 0.5 to 0.7 significant discrepancies, 0.2 to 0.4 major errors, 0.0 to 0.1 complete mismatch). Success rate is reported separately.
Cost units
The table lists each model's list price per million input and output tokens. It does not publish total run cost.
Time units
Average time per generation in milliseconds, as published in the table.
No comments yet.