Tickfloor

Multiple testing

Last updated

Multiple testing is the problem that the more rules you try, the more likely one looks good by chance. A result has to be judged against how many were tried.

How multiple testing is calculated

If you run 20 independent tests at a 5 percent significance level, you expect about one false positive. Corrections raise the bar: Bonferroni divides the threshold by the number of tests, and the Benjamini-Hochberg method controls the false discovery rate. The Deflated Sharpe Ratio adjusts a Sharpe ratio for the number of trials.

How it is read

A result that was the best of many tries needs a stronger test than one specified in advance.

Common mistakes

  • Counting only the tests you kept.
  • Treating small changes to parameters as one test. Each is a try.
  • Reading an unadjusted p-value from the best of many.

Test it yourself

The Lab counts attempts on each experiment and shows the count on the receipt, so the number of tries stays visible. The Lab charges trading costs on every trade, fills on the next day's open, and shows how many attempts you have made.

What Tickfloor tested

1,346strategies tested across all our research
0passed the luck check
How we counted

Tickfloor's research desk has backtested 1,346 strategies, net of modelled trading costs. After correcting for the 1,778 tests run, 0 passed. That does not prove no strategy works. These are results from our older test setup, which was built before our current testing rules. They have not been redone under those rules.

Tests on Multiple testing

Lessons that cover it

Related concepts

General information only. It doesn't consider your objectives, finances or needs. Tickfloor holds no financial services licence and never places trades.

Keep reading