Overfitting
Last updated
Overfitting is fitting a rule so closely to past data that it captures noise instead of anything repeatable. It looks excellent on the history it was tuned on and disappoints afterwards.
How overfitting is spotted
There is no single measure. Signs: many tunable parameters relative to the number of trades, a result that depends on one exact setting, and a large gap between the result on the tuning data and on data held back. The Deflated Sharpe Ratio and the probability of backtest overfitting are two published ways to put a number on it.
How it is read
A rule with two parameters and a stable result across nearby settings is less likely to be overfit than one with ten and a single sharp peak.
Common mistakes
- Tuning on all the data and calling it a test.
- Adding a filter each time a loss appears, until the history is explained.
- Reporting the best of many tries without counting the tries.
What Tickfloor tested
How we counted
Tickfloor's research desk has backtested 1,346 strategies, net of modelled trading costs. After correcting for the 1,778 tests run, 0 passed. That does not prove no strategy works. These are results from our older test setup, which was built before our current testing rules. They have not been redone under those rules.
Tests on Overfitting
Lessons that cover it
Related concepts
General information only. It doesn't consider your objectives, finances or needs. Tickfloor holds no financial services licence and never places trades.