Overfitting

Overfitting is fitting a rule so closely to past data that it captures noise instead of anything repeatable. It looks excellent on the history it was tuned on and disappoints afterwards.

How Overfitting is calculated

There is no single measure. Signs: many tunable parameters relative to the number of trades, a result that depends on one exact setting, and a large gap between the result on the tuning data and on data held back. The Deflated Sharpe Ratio and the probability of backtest overfitting are two published ways to put a number on it.

How it is read

A rule with two parameters and a stable result across nearby settings is less likely to be overfit than one with ten and a single sharp peak.

Common mistakes

• Tuning on all the data and calling it a test.
• Adding a filter each time a loss appears, until the history is explained.
• Reporting the best of many tries without counting the tries.

What Tickfloor tested

Tickfloor's research desk has backtested 678 strategies, net of modelled trading costs. After correcting for the 761 tests run, 0 passed. That does not prove no strategy works. These are historical diagnostics, not validation under Testing Standard v2: the backtest harness predates that standard and has not been re-run to meet it.

• How lucky is our best result? (study)

Lessons that cover it

• Luck or skill: coin flips, fund managers and convincing charts
• Why testing many ideas finds lucky ones
• Going live and checking it still works
• Preregistration: write the test down before you look

• Short definition in the glossary

Related concepts

• Multiple testing
• Out-of-sample testing
• Walk-forward analysis
• Backtest
• All concepts

General information only. It doesn't consider your objectives, finances or needs. Tickfloor holds no financial services licence and never places trades.

• See a full worked example (free, no account)
• Browse every course lesson by topic