Backtesting historic results have changed

[naples99]

I have been running a strategy for the last 2-3 months and been keeping a separate note of trade entry/exit times and prices based on signals generated by my strategy just in case historic signals were 'rewritten' by the backtester. I've run this program countless times and it has always produced the same results until this morning. When I ran the strategy through the backtester this morning I got slightly different entry and exit times and therefore different trade prices. Obviously this is slightly concerning.

After speaking with the heldesk I installed the latest version of eSignal and the results are different again (i.e. neither of the 2 previous set of results) for the period of the last 2 weeks although up until that point they appear to be what I had originally expected.

I'd be grateful if anyone could anyone shed some light on what might be going on here (I accept that it could be something I've done but I hadn't made any changes to the code or time templates, interval etc so am struggling to find any answers/reasons) because if this is a regular occurance it throws the whole point of using esignal as a platform to generate a starategy into doubt.

The product in question is the DAX i.e. AZ 1!-DT.

Many thanks in advance

[ACM]

Re: Backtesting historic results have changed

naples99
The behavior your describe could happen depending on the studies being used to generate the signals (as an example see this thread in which you reported a similar behavior)
However without being able to backtest the same script you are using it is not possible to determine if this is the issue or if it is caused by some other problem
Alex

Originally posted by naples99
I have been running a strategy for the last 2-3 months and been keeping a separate note of trade entry/exit times and prices based on signals generated by my strategy just in case historic signals were 'rewritten' by the backtester. I've run this program countless times and it has always produced the same results until this morning. When I ran the strategy through the backtester this morning I got slightly different entry and exit times and therefore different trade prices. Obviously this is slightly concerning.

After speaking with the heldesk I installed the latest version of eSignal and the results are different again (i.e. neither of the 2 previous set of results) for the period of the last 2 weeks although up until that point they appear to be what I had originally expected.

I'd be grateful if anyone could anyone shed some light on what might be going on here (I accept that it could be something I've done but I hadn't made any changes to the code or time templates, interval etc so am struggling to find any answers/reasons) because if this is a regular occurance it throws the whole point of using esignal as a platform to generate a starategy into doubt.

The product in question is the DAX i.e. AZ 1!-DT.

Many thanks in advance

[naples99]

Hi Alex,

Once again thanks for a very prompt reply.

I had that similar problem in mind when I encountered this one. Since then I have masde sure that I have run my backtests on ALL the intra-day data available (in this case back to 13/3/07) so that I have a fixed start point.

Also just to re-iterate, it's not that I'm getting different results day to day it's that I have suddenly got different results while using the same dataset. Is there any way to get the indicator in question to you without posting it on a public forum.

[naples99]

Today I started up eSignal and the indicator is again showing the results I was getting yesterday's pre-update (i.e. they are still different to what I originally expected). This is very confusing. Why, when it appears nothing has changed, is this sort of thing happening? And also why would installing the latest upgrade cause a temporary change in my results for just one day?

[naples99]

Here is the code I am using and getting strange results for.

I run it on all intra-day data available on a 1 min interval.

The problem is with AX 1!-DT. I haven't taken as detailed historic paper notes on other products.

If you need any other info Alex please ask

Many thanks

/*********************************
Provided By:
eSignal (Copyright c eSignal), a division of Interactive Data
Corporation. 2008. All rights reserved. This sample eSignal
Formula Script (EFS) is for educational purposes only and may be
modified and saved under a new file name. eSignal is not responsible
for the functionality once modified. eSignal reserves the right
to modify and overwrite this EFS file with each new release.

Description:
ECO (Blau`s Ergodic Candlestick Oscillator)

Version: 1.0 12/03/2008

Formula Parameters: Default:
Number To Calculate EMA 32
Number To Calculate vEMA 12

Notes:
We call this one the ECO for short, but it will be listed on the indicator list
at W. Blau’s Ergodic Candlestick Oscillator. The ECO is a momentum indicator.
It is based on candlestick bars, and takes into account the size and direction
of the candlestick "body". We have found it to be a very good momentum indicator,
and especially smooth, because it is unaffected by gaps in price, unlike many other
momentum indicators.
We like to use this indicator as an additional trend confirmation tool, or as an
alternate trend definition tool, in place of a weekly indicator. The simplest way
of using the indicator is simply to define the trend based on which side of the "0"
line the indicator is located on. If the indicator is above "0", then the trend is up.
If the indicator is below "0" then the trend is down. You can add an additional
qualifier by noting the "slope" of the indicator, and the crossing points of the slow
and fast lines. Some like to use the slope alone to define trend direction. If the
lines are sloping upward, the trend is up. Alternately, if the lines are sloping
downward, the trend is down. In this view, the point where the lines "cross" is the
point where the trend changes.
When the ECO is below the "0" line, the trend is down, and we are qualified only to
sell on new short signals from the Hi-Lo Activator. In other words, when the ECO is
above 0, we are not allowed to take short signals, and when the ECO is below 0, we
are not allowed to take long signals.
**********************************/

/*********************************


**********************************/

var fpArray = new Array();
var bInit = false;
var ECO = 0.0;
var ECOprev = 0.0;

function preMain() {
setStudyTitle("XYZ");
setCursorLabelName("XYZ", 0);
setDefaultBarFgColor(Color.red, 0);
addBand(0, PS_SOLID, 1, Color.cyan);

var x=0;
fpArray[x] = new FunctionParameter("r", FunctionParameter.NUMBER);
with(fpArray[x++]){
setLowerLimit(1);
setDefault(960);
}
fpArray[x] = new FunctionParameter("s", FunctionParameter.NUMBER);
with(fpArray[x++]){
setLowerLimit(1);
setDefault(360);
}
}

var xEMA = null;
var xvEMA = null;

function main(r, s) {

var nState = getBarState();

if (nState == BARSTATE_ALLBARS) {
if (r == null) r = 960;
if (s == null) s = 360;
}

if ( bInit == false ) {
xEMA = ema(s, ema(r, efsInternal("CloseOpen")));
xvEMA = ema(s, ema(r, efsInternal("HighLow")));
bInit = true;
}

if (getCurrentBarCount() < Math.max(r,s)) return;

if(xvEMA.getValue(0) != 0) {
ECO = 100 * (xEMA.getValue(0) / xvEMA.getValue(0));
}



if(xvEMA.getValue(0) != 0) {
ECOprev = 100 * (xEMA.getValue(0) / xvEMA.getValue(0));
}

if(xvEMA.getValue(0) != 0)

return 100 * (xEMA.getValue(0) / xvEMA.getValue(0));


else
return;
}

function CloseOpen() {
var nRes = 0;
nRes = close(0) - open(0);
if (nRes == null) nRes = 1;
return nRes;
}

function HighLow() {
var nRes = 0;
nRes = high(0) - low(0);
if (nRes == null) nRes = 1;
return nRes;
}

[naples99]

This is still an ongoing problem. I'd really appreciate any help with this.

Yesterday my results reverted to what I had originally expected. Today my indicator has again produced the erroneous set from a few days ago. At least I am calling them erroneous because if this continues to happen I have no idea which set of results is the 'correct' set.

I have gone through all the usual reason why this may happen (interval, time template, start point etc etc) and have no idea why my results (i.e. time and price of trades) seem to flip back a forth from day to day between 2 (and once a third) sets of results when all the parameters remain the same - or at least to my best knowledge they are.

Many thanks in advance for any suggestions as to why this could be happening.

[DolmenProp]

Hi,

I don't know if the problem is still relevent to you however I have encountered the same problem a couple of times and thought I'd share my 2 cents.

I think the problem lies with the fact that you are using an EMA (the problem extends to the MACD&ADX and any other indicator that takes an EMA of something)

The problem is a mathematical one relating to iterative functions.
This is when todays value of the function depends on yesterdays, which an EMA does. When you calculate an EMA (9 period say), you have to take a normal MA for the first 9 bars because there is no 'yesterday' to calculate the EMA. This makes the function path dependent over the lifetime of the strategy.

The difference between this and say a 9 period RSI is that after 9 bars, the RSI has no knowledge of where the strategy began, i.e it is not path dependent for the WHOLE strategy, only for 9 bar sections of it.

Now, for a non-program based trader this doesn't make a difference, as the compounded difference is minuscule. When you use a program to define trade signals, you may encounter a case where a crossover (say of the two MACD lines) just about happens today, but wouldn't if you started the strategy another day. If your strategy then depends on if you're long/short to execute another signal, the compounded affect can be very large.

The way I get around this is by sticking to a start date for a time template and then every day add 1 bar to it. It's a bit of a pain but the only way I can think of to get around it.

Anyway, hope it helps. Some people dont like sharing info on forums but i'm here to get things right and if enough of us contribute, we'll be printing money in no time!

John