Varying results via backtesting

[takis]

using the same formula on the same time templates gives me different results!
that thing almost drove me crazy!
i sat down and checked each and every trade for the same period(for example 4/2) and what do i get?
the trades generated are correct!
but this mornig while the trades were generated in real time they were DIFFERENT ones! (i had marked them down on the paper)
so the data feed changes!
and the candles i was getting this morning with all the studies on them have different prices tonite!

example:
today on 15.05 Cet there is a 5' candle on dax future with prices for the OHLC and prices for the macd(12,26,9)
while the candle was forming the macd crossed and stayed crossed so, i got a short signal generated by my formula.
tonite, i go to do an overview of my trading day and what do i see?
on the same candle the same time the macd is NOT crossed and the formula as a result does not generate a trade!!!

as a result the backtesting cannot be reliable as who knows if the trades that gives actually happened the previous days?

so, testing my system 3 hours ago from 19/12 till today it gives me 150 points profit and now that no trading has happend meanwhile it gives me 215 points !!!
the trades taken one by one before where correct but on the other hand so are the ones taken now!

what do i have to do?
does anybody know?
has that thing happened to you?

[Doji3333]

I could be wrong....

But I would suspect this is an issue with "Real-Time Actions" vs. "Historical Actions".

I have seen this over and over with different systems. In fact, most of the time, I have to code things differently for historical vs. real-time activities.

The results could simply be the fact that in Real-time, certain actions were taken based on your indicators while on a historical bases (after the bar completely forms) the indicator may have reacted differently. I know stochastics reacts this way (in RT vs. Historical). So I would think you MACD may be doing the same thing.

What is the solution???

You can't replicate real-time data in a back-test. There are inherent differences that you can't overcome (the biggest is that bars appear to form as ONE COMPLETE BAR - not tick by tick). Many many people of the years have had this same problem.

The best thing you can do is design your system to act "as close to RT as possible"...

Let me know if I can help..

B

[takis]

i do not really think this is the problem cause my system is made so i get the signal when the bar has allready formed.
for example i get a signal to enter the trade at the open of the next candle AFTER the previous one has formed. so i believe the data stay the same.
its not actually a real time problem i think.
for example if i backtest the days fro 19/12 till 02/02 i got different results !!
and those were historical data now and 3 hours ago!
i mean that the same PREVIOUS days give different results when backtested with the same system now and this morning or yesterday.
do you have an idea for that?

[eSignal_JasonK]

Brad, you’re quick on the draw and excellent response as well.

Takis, Brad pretty well explained things, but since I already had this typed up I thought I’d share my response anyway. Let us know if you have more questions.

The Strategy object is only intended for back testing formulas through the Strategy Analyzer, not for real time analysis. When back testing, the formula only processes on historical bars and only sees the Open, High, Low and Close price of the bar. It does not take into consideration what would have happened during the interval's intra-bar trades. So if you are using Close as the price source in your MACD study to trigger an Entry the Strategy Analyzer does not automatically make the assumption that there would have been a cross intra-bar at the low of the bar. It only sees what the values for the MACD are at the close of the bar, where there may not be a cross. Another factor that will cause a difference in results for back testing vs. real time analysis is the parameters used with the Strategy.doLong() and .doShort() calls for the Fill Type and Fill Bar. If you use Strategy.MARKET / Strategy.NEXTBAR, you will may get a different fill price than the real time scenario. Also, make sure you aren’t using Strategy.MARKET / Strategy.THISBAR combo as that will give you a fill price at the open of the bar in question, which may not be a realistic fill price. At any rate, the Strategy Analyzer will not produce the exact same results over a specified period of time when compared to real time trading results over that same period of time, but it should give you a good idea of how well a strategy would have performed. I personally recommend keeping real time formulas and back testing formulas completely separate for this reason. Once you're satisfied with how a strategy back tests, then write a new formula and track your trades with regular JavaScript variables (i.e. vPosition == "L" vs. Strategy.isLong() == true etc.) and use the paper trader to do some real time testing and fine tuning.

[ScottJ]

One other item to add here is that it's possible that you may be hitting different history servers as you test over time. This FAQ explains the situation in greater detail and as it indicates towards the bottom of the FAQ, we are about to implement a new synchronization process to make all history servers stamp at the same time.

This issue occurs as you make historical requests and get connected to servers that are ever so slightly off one another in terms of time-stamping. It shouldn't be an issue in real-time as the data streams in.

Thanks.

[eSignal_JasonK]

takis,

Did the back test you ran 3 hours ago include testing on data after 2/2? And did this back test also start at the same bar as the test you ran previous to that? Can you post your code so we can do some testing?

[takis]

sure.
i think Scott is closer to the problem.
here is my code.
test it today and try to test it later or tommorow on 5' on AX H4-DT

yes. i make sure the system is tested for the same templates
even in periodical analysis on daily analysis i get different results for some days, not all of them

[Guest]

One other thing that I have always known, but has come to impact me more recently when going through backtest development. The computer is only doing what you have programmed it to do.

Both Brad and Jason hit on that aspect. It is fairly easy to write simple strategies to perform backtesting which will help you evaluate a trading system. It is much more difficult to write one which models and works in real time. To do that you must understand the limitations of what you are doing and the tools you are using. Both Brad and Jason know that very well. I am learning.

eSignal efs is very powerful and can be written to implement an infinite number of extremely complex analysis. I would suggest that you try and understand better what Brad and Jason are saying.

For what it is worth...

[takis]

you are right on that one.
the only way to be sure is to paper trade the system in real time and thats what i do anyway, with an excel i have made testing 19 systems simutaniously in real time as they come up.
but the thing is that when a closed bar on real time has given 2 number for macd (lets say 3,08 and 3,10) then i enter a trade on the NEXT candle.
the problem is that when I go back at the end of the day to see that trade on the backtester, it has not been generated as the macd on the same candle of the chart has prices 3,09 and 3,10)
that is a problem and from what i see of Scott it should be the servers.

i understand what you say.
sometimes while in real time the signal appears and dissapears during the formation of the candle.
that is something else and different prices for the same PREVIOUS formated bar is another.

that is the reason why my system generetes a signal ONLY if the previous candle has CLOSED meeting my parameters not during its formation

[eSignal_JasonK]

Hello takis,

Thanks for posting your code. I set up a 2-day time template and ran your formula with a 5 min interval. Here's the results, which cover the trading hours of 6:30 - 11:00 PST for 2/3-2/4:



If I use the same time template tomorrow evening, the back test will only cover 2/4-2/5. These two tests will not be the same. Is this the cause of the difference you are experiencing? If you are talking about a difference in a fill price on the same bar and day from two different back tests, then you are most likely experiencing the issue that Scott has outlined.

[takis]

please do me a favor and test the formula for 36 days template using 1 contract as a default and post the results.
so we can compare them

[takis]

[eSignal_JasonK]

Hi Takis,

Here's a 36-day time template, no commission, no slippage and 1 contract per trade. The time template covers 00:00 - 11:00 PST.



Here are the beginning trades.

[takis]

i guess you did not change the time in the code as it is set up for my time difference (+1.00 cet)
and its from when trading begins on dax till 10 minutes before the close of it.
if you do that you should get these results:

[takis]

and this sample for January and feb: