What our tested strategies show
We have tested 12 625 trading ideas on 8 markets, 454 602 results in total. Here is what the data shows, including the uncomfortable parts. Calculated on the whole history (ProfitFeed before IG, adjusted data) and with at least 30 trades per result.
A library of ideas, not a recommendation. The figures apply to automatically generated ideas, most of which are deliberately simple variants. Applies to CFD and spread betting. A basis for your own studies.
1. Most strategies do not make money
Of 295 374 combinations of strategy and market only 24.7 % made a profit over the whole history. Only 0.2 % gave a higher total return than simply owning the index, and 3.5 % beat the index on return per drawdown.
| Market | Made a profit | Beat the index (return per drawdown) |
|---|---|---|
| Nasdaq 100 | 38.2 % | 4.0 % |
| Nikkei 225 | 29.6 % | 1.1 % |
| Wall Street (Dow) | 28.1 % | 6.1 % |
| S&P 500 | 27.4 % | 3.6 % |
| DAX 40 | 22.6 % | 4.7 % |
| Gold | 20.3 % | 3.2 % |
| FTSE 100 | 6.9 % | 1.6 % |
EUR/USD stands out: the currency barely moved during the period, so the bar for beating the index is low there. On equity indices and gold, which have risen sharply, it is much harder. That is why we always report the comparison with the index.
2. The best on training data fall back
We split each history into training (oldest), validation and test (newest). If you pick only the best on the training part you see how much was luck.
| Group | Profit factor training | Profit factor validation | Profit factor test | Profit in the test part | Profit in both validation and test |
|---|---|---|---|---|---|
| All strategies 351 317 strategies | 0.94 | 0.91 | 0.88 | 39.1 % | 17.6 % |
| Best 10 % in training 35 131 strategies | 1.66 | 1.04 | 1.01 | 50.5 % | 28.4 % |
| Best 1 % in training 3 513 strategies | 2.80 | 0.96 | 1.13 | 54.7 % | 30.7 % |
The best 1 % on training data had a median profit factor of 2.80. In validation it was 0.96 and in the test 1.13: a sharp fall, but still above break-even. 30.7 % made a profit in both validation and test, so roughly every other top strategy holds up on new data. A training figure should never be taken as truth.
How we reduce the risk of being fooled by chance
- Three-way split: search only on training, choose the configuration before validation is opened, touch the test part last.
- A plateau, not a peak: a setting whose neighbours are poor is curve-fitted.
- Account for multiple testing: hundreds of combinations give a few that look good by pure chance.
- Costs from the start: spread and fees are included in the search.
- Open reporting: strategies that do not hold up are shown too, compared with simply owning the index.