Announcement

Collapse
No announcement yet.

Backtesting historic results have changed

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • Backtesting historic results have changed

    I have been running a strategy for the last 2-3 months and been keeping a separate note of trade entry/exit times and prices based on signals generated by my strategy just in case historic signals were 'rewritten' by the backtester. I've run this program countless times and it has always produced the same results until this morning. When I ran the strategy through the backtester this morning I got slightly different entry and exit times and therefore different trade prices. Obviously this is slightly concerning.

    After speaking with the heldesk I installed the latest version of eSignal and the results are different again (i.e. neither of the 2 previous set of results) for the period of the last 2 weeks although up until that point they appear to be what I had originally expected.

    I'd be grateful if anyone could anyone shed some light on what might be going on here (I accept that it could be something I've done but I hadn't made any changes to the code or time templates, interval etc so am struggling to find any answers/reasons) because if this is a regular occurance it throws the whole point of using esignal as a platform to generate a starategy into doubt.

    The product in question is the DAX i.e. AZ 1!-DT.

    Many thanks in advance
    Last edited by naples99; 01-11-2010, 04:26 AM.

  • #2
    Re: Backtesting historic results have changed

    naples99
    The behavior your describe could happen depending on the studies being used to generate the signals (as an example see this thread in which you reported a similar behavior)
    However without being able to backtest the same script you are using it is not possible to determine if this is the issue or if it is caused by some other problem
    Alex


    Originally posted by naples99
    I have been running a strategy for the last 2-3 months and been keeping a separate note of trade entry/exit times and prices based on signals generated by my strategy just in case historic signals were 'rewritten' by the backtester. I've run this program countless times and it has always produced the same results until this morning. When I ran the strategy through the backtester this morning I got slightly different entry and exit times and therefore different trade prices. Obviously this is slightly concerning.

    After speaking with the heldesk I installed the latest version of eSignal and the results are different again (i.e. neither of the 2 previous set of results) for the period of the last 2 weeks although up until that point they appear to be what I had originally expected.

    I'd be grateful if anyone could anyone shed some light on what might be going on here (I accept that it could be something I've done but I hadn't made any changes to the code or time templates, interval etc so am struggling to find any answers/reasons) because if this is a regular occurance it throws the whole point of using esignal as a platform to generate a starategy into doubt.

    The product in question is the DAX i.e. AZ 1!-DT.

    Many thanks in advance

    Comment


    • #3
      Hi Alex,

      Once again thanks for a very prompt reply.

      I had that similar problem in mind when I encountered this one. Since then I have masde sure that I have run my backtests on ALL the intra-day data available (in this case back to 13/3/07) so that I have a fixed start point.

      Also just to re-iterate, it's not that I'm getting different results day to day it's that I have suddenly got different results while using the same dataset. Is there any way to get the indicator in question to you without posting it on a public forum.

      Comment


      • #4
        Today I started up eSignal and the indicator is again showing the results I was getting yesterday's pre-update (i.e. they are still different to what I originally expected). This is very confusing. Why, when it appears nothing has changed, is this sort of thing happening? And also why would installing the latest upgrade cause a temporary change in my results for just one day?

        Comment


        • #5
          Here is the code I am using and getting strange results for.

          I run it on all intra-day data available on a 1 min interval.

          The problem is with AX 1!-DT. I haven't taken as detailed historic paper notes on other products.

          If you need any other info Alex please ask

          Many thanks

          /*********************************
          Provided By:
          eSignal (Copyright c eSignal), a division of Interactive Data
          Corporation. 2008. All rights reserved. This sample eSignal
          Formula Script (EFS) is for educational purposes only and may be
          modified and saved under a new file name. eSignal is not responsible
          for the functionality once modified. eSignal reserves the right
          to modify and overwrite this EFS file with each new release.

          Description:
          ECO (Blau`s Ergodic Candlestick Oscillator)

          Version: 1.0 12/03/2008

          Formula Parameters: Default:
          Number To Calculate EMA 32
          Number To Calculate vEMA 12

          Notes:
          We call this one the ECO for short, but it will be listed on the indicator list
          at W. Blau’s Ergodic Candlestick Oscillator. The ECO is a momentum indicator.
          It is based on candlestick bars, and takes into account the size and direction
          of the candlestick "body". We have found it to be a very good momentum indicator,
          and especially smooth, because it is unaffected by gaps in price, unlike many other
          momentum indicators.
          We like to use this indicator as an additional trend confirmation tool, or as an
          alternate trend definition tool, in place of a weekly indicator. The simplest way
          of using the indicator is simply to define the trend based on which side of the "0"
          line the indicator is located on. If the indicator is above "0", then the trend is up.
          If the indicator is below "0" then the trend is down. You can add an additional
          qualifier by noting the "slope" of the indicator, and the crossing points of the slow
          and fast lines. Some like to use the slope alone to define trend direction. If the
          lines are sloping upward, the trend is up. Alternately, if the lines are sloping
          downward, the trend is down. In this view, the point where the lines "cross" is the
          point where the trend changes.
          When the ECO is below the "0" line, the trend is down, and we are qualified only to
          sell on new short signals from the Hi-Lo Activator. In other words, when the ECO is
          above 0, we are not allowed to take short signals, and when the ECO is below 0, we
          are not allowed to take long signals.
          **********************************/

          /*********************************


          **********************************/

          var fpArray = new Array();
          var bInit = false;
          var ECO = 0.0;
          var ECOprev = 0.0;

          function preMain() {
          setStudyTitle("XYZ");
          setCursorLabelName("XYZ", 0);
          setDefaultBarFgColor(Color.red, 0);
          addBand(0, PS_SOLID, 1, Color.cyan);

          var x=0;
          fpArray[x] = new FunctionParameter("r", FunctionParameter.NUMBER);
          with(fpArray[x++]){
          setLowerLimit(1);
          setDefault(960);
          }
          fpArray[x] = new FunctionParameter("s", FunctionParameter.NUMBER);
          with(fpArray[x++]){
          setLowerLimit(1);
          setDefault(360);
          }
          }

          var xEMA = null;
          var xvEMA = null;

          function main(r, s) {

          var nState = getBarState();

          if (nState == BARSTATE_ALLBARS) {
          if (r == null) r = 960;
          if (s == null) s = 360;
          }

          if ( bInit == false ) {
          xEMA = ema(s, ema(r, efsInternal("CloseOpen")));
          xvEMA = ema(s, ema(r, efsInternal("HighLow")));
          bInit = true;
          }

          if (getCurrentBarCount() < Math.max(r,s)) return;

          if(xvEMA.getValue(0) != 0) {
          ECO = 100 * (xEMA.getValue(0) / xvEMA.getValue(0));
          }



          if(xvEMA.getValue(0) != 0) {
          ECOprev = 100 * (xEMA.getValue(0) / xvEMA.getValue(0));
          }

          if(xvEMA.getValue(0) != 0)

          return 100 * (xEMA.getValue(0) / xvEMA.getValue(0));


          else
          return;
          }

          function CloseOpen() {
          var nRes = 0;
          nRes = close(0) - open(0);
          if (nRes == null) nRes = 1;
          return nRes;
          }

          function HighLow() {
          var nRes = 0;
          nRes = high(0) - low(0);
          if (nRes == null) nRes = 1;
          return nRes;
          }
          Last edited by naples99; 01-12-2010, 07:33 AM.

          Comment


          • #6
            This is still an ongoing problem. I'd really appreciate any help with this.

            Yesterday my results reverted to what I had originally expected. Today my indicator has again produced the erroneous set from a few days ago. At least I am calling them erroneous because if this continues to happen I have no idea which set of results is the 'correct' set.

            I have gone through all the usual reason why this may happen (interval, time template, start point etc etc) and have no idea why my results (i.e. time and price of trades) seem to flip back a forth from day to day between 2 (and once a third) sets of results when all the parameters remain the same - or at least to my best knowledge they are.

            Many thanks in advance for any suggestions as to why this could be happening.

            Comment


            • #7
              Hi,

              I don't know if the problem is still relevent to you however I have encountered the same problem a couple of times and thought I'd share my 2 cents.

              I think the problem lies with the fact that you are using an EMA (the problem extends to the MACD&ADX and any other indicator that takes an EMA of something)

              The problem is a mathematical one relating to iterative functions.
              This is when todays value of the function depends on yesterdays, which an EMA does. When you calculate an EMA (9 period say), you have to take a normal MA for the first 9 bars because there is no 'yesterday' to calculate the EMA. This makes the function path dependent over the lifetime of the strategy.

              The difference between this and say a 9 period RSI is that after 9 bars, the RSI has no knowledge of where the strategy began, i.e it is not path dependent for the WHOLE strategy, only for 9 bar sections of it.

              Now, for a non-program based trader this doesn't make a difference, as the compounded difference is minuscule. When you use a program to define trade signals, you may encounter a case where a crossover (say of the two MACD lines) just about happens today, but wouldn't if you started the strategy another day. If your strategy then depends on if you're long/short to execute another signal, the compounded affect can be very large.

              The way I get around this is by sticking to a start date for a time template and then every day add 1 bar to it. It's a bit of a pain but the only way I can think of to get around it.

              Anyway, hope it helps. Some people dont like sharing info on forums but i'm here to get things right and if enough of us contribute, we'll be printing money in no time!

              John

              Comment

              Working...
              X