Explore more topics
Worked strategy research

Prevent overfitting: decide what passes before the test

Follow a real NVDA experiment: set requirements, freeze the chosen rules, inspect later results and keep the failed requirements visible.

As of 2026-10-10

Jump to a section

TL;DR

Write acceptance requirements before testing and keep them fixed even when the selected candidate makes money. This NVDA version returned 22.06% on April to June 2024 dates but failed four of five checks: it lagged the 36% stored NVDA benchmark, exceeded 10% drawdown, completed no round trips and held overnight. Freeze the rules, search budget and data roles, then inspect signals and holdings against the intended task. Published dates are now reproduction evidence; choose untouched dates before developing another candidate.

Five predeclared checks rejected this profitable candidate

Animation: an AI-tuned NVDA bot that returned 22.06% in its April to June 2024 test passes one rule and fails four: it did not beat NVDA's 36.00%, its worst drop was 12.73%, it completed 0 round trips, and it held for three months. It is stamped rejected.
The bot made 22.06% but failed four of the five rules written before the test, so it was not promoted.
Show the table · 5 rows, 3 columns

Scroll sideways to compare all columns. Select a heading to sort.

Positive later return22.06%Pass
Beat the stored NVDA buy-and-hold return22.06%Fail
Maximum drawdown at most 10%12.73%Fail
At least 10 closed round trips0Fail
No position held overnightOne buy, no sells; position held to test endFail

All five requirements had to pass. The NVDA benchmark threshold was 36.00%; the drawdown ceiling was 10%; the activity minimum was 10 closed round trips.

The candidate's 22.06% later return passed one requirement and failed four. The original acceptance criteria and measured results are retained here. Keeping them fixed prevented a profitable buy-and-hold-like result from being promoted as the requested intraday strategy.

Freeze a protocol that can reject the winner

Show the table · 5 rows, 3 columns

Scroll sideways to compare all columns. Select a heading to sort.

Define the taskEntry family, allowed mutations, sizing, exits and intraday holding requirementThe selected exit became $378,389.83; inspect the complete rules, not the seed name
Declare the search budgetPopulation, generations, windows and every restartThis search used population 6, 3 generations and 1 training window; six final rows do not count every evaluated variant
Separate data rolesTraining for development; validation for selection; a later frozen test for acceptanceValidation selected the inactive 0% candidate; April–June then tested the frozen configuration
Preserve failure criteriaReturn versus same-period baseline, drawdown, closed-trade count and overnight holdingsDo not raise 10% drawdown or remove 10-round-trip/no-overnight requirements after seeing the result
Protect the next testLater dates excluded from every candidate revision and corpus queryThe published 2024 dates are now a reproduction window; choose untouched dates before developing a new variant

Inspect the frozen later test

Portfolio value (USD)

NVDA genetic winner, original morning strategy, NVDA buy and hold, and SPY buy and hold: April to June 2024$15.5K$8.3KLargest decline in plotted observations: 12.40%, 2024-06-20, 04:01:00 EDT to 2024-06-25, 04:01:00 EDT2024-04-012024-06-28
Genetic winner 2024-06-28, 20:00:00 EDT
$12,205.98
Original morning strategy 2024-06-28, 20:00:00 EDT
$8,980.94
NVDA buy and hold 2024-06-28, 20:00:00 EDT
$13,600.01
SPY buy and hold 2024-06-28, 20:00:00 EDT
$10,371.61

Genetic winner: 12.40% largest decline in plotted observations. 2024-06-20, 04:01:00 EDT ($13,328.24) to 2024-06-25, 04:01:00 EDT ($11,675.87).

Genetic winnerOriginal morning strategyNVDA buy and holdSPY buy and holdHistorical simulationDates shown in New York time

The selected candidate earned 22.06% in the later test. The stored order record contains one filled buy and no filled sells, so this positive result came with an overnight position and zero completed round trips. Compare the winner, unchanged morning seed and the stored NVDA/SPY baselines on their shared later dates.

Verify the accepted signal became the intended trading behavior

Inspect the exact historical measurements

All four recorded entry conditions were true at April 1, 13:30 UTC. The complete order query returned one buy of 66.015730 NVDA shares at $90.30 with a $2.98 fee, and no filled sells. End holdings were $8,169.45 beside $4,036.53 cash. The planned no-overnight and closed-trade checks therefore matter independently of the 22.06% return.

Scroll sideways to compare all columns. Select a heading to sort.

Minutes after open0≤0True
Minutes after open0≤139True
NVDA price / SMA90.29900360107422=90.29900360107422True
Position value0≥0True

Continue exploring

Discussion

Sign in or create a free account to join the discussion.

No comments yet.

Aurora · AI research assistant

Try this in Aurora

Edit the request, then open it in Aurora. You choose when to send it.