the enigma that is backtesting

[treblewide]

folks,

i am looking for input and experiences of people who seriously backtest systems.

The reason this has come about is as follows.....me and a group of fellow traders have been developing a system which in 2004 has produced a 350% return so far...it is amazingly good..tiny drawdowns (12% max). there have been a few thoughts going through my head..this is it, this is it!!!!!!

however..the problem....on historical backtesting we have found that in 2001 it lost quite a bit of money, 2002 it made a reasonable profit, 2003 it was flat but 2004 has produced about 300 full s+p points so far......

I know a system needs to stand up to testing over a longer period however what is the optimum time frame for backtesting on an instrument that is ever changing like the e-mini s+p and how do you keep fitting a system into current market conditions?

Our problem is why optimise a system to take into account the craziness of 2001 when we may never get a year of volatility like it??

experiences, thoughts, views welcome!

Treblewide

[Momentum]

Treblewide, I think that perhaps you are missing the point a bit. I don't want to pontificate but you deserve an honest answer.

You ALWAYS get "some sort of craziness", but a different sort each time. Systems usually fall into one of two categories - trend following or trend fading. If you have enough of the type of market that your system is designed for and have GOOD money management rules, you make money. With diversification, you increase your odds and reduce your drawdowns.

I don't want to disappoint byou, but most systems can make money with good money management rules. Its a matter of whether:
1. you have the discipline to follow the system as designed, and
2. you have pockets deep enough to ride out the drawdowns.

Momentum

[moebot]

Assuming you are using daily data, I would recommend testing at least 10 years back to get a good idea if your system is robust enough to handle changing market conditions. Volatility is low now, but volatility itself tends to move in long term cycles. Your system should be able to handle these cycles.

From your earlier post, I'd say you've fit a nice oscillating system to the data, but you won't do well in a trending market. What is interesting is that this is the mirror opposite of a trend following system, ie turtles, etc. That you were flat in 2003 when every index in the US showed huge gains is telling.

If you were able to combine your system with a trend following component, you may be able to profit from both types of markets.

PS. FWIW, your system is likely to work well through the elections.