Changing linear regression values

[hamidkassam]

I am trying to make use of linear regression values in a trading strategy. The original formula study was copied from an efs posted by Alexis.

The problem I am having is that when the efs is run on realtime data, I get one set of values. But when I run it on historical data (or when I refresh the data) the values change. This is despite using the SetComputeOnClose() statement. What am I missing?

I would like the values in realtime to be the same as when the efs is applied on historical data.

Here is the efs file in question followed by examples of different values under the two scenarios. Any help from anyone would be greatly appreciated.

/*
Test of Linear regression values 
Formula from an example study by Alexis Montenegro
*/

var sum                 = null;
var LR                  = null;
var LR1                 = null;
var LR2                 = null;
var LR3                 = null;
var Length              = 25;
var bInitialized        = false;
var aFPArray            = new Array();
var Time                = null;
var sMkt                = null;

function preMain() { 
    setPriceStudy(true);
    setStudyTitle("Linear Regression"); 
    setCursorLabelName("LR");
    setDefaultBarThickness(3);
    setPlotType(PLOTTYPE_INSTANTCOLORLINE);
    setComputeOnClose();
    
    x=0;
    aFPArray[x] = new FunctionParameter( "bTest", FunctionParameter.BOOLEAN);
    with( aFPArray[x] ) {
        setName( "Start Test?" );
        addOption( true );    
        addOption( false );    
        setDefault( true );
    }    
    
}

function main(bTest) {

    if ( getBarState() == BARSTATE_ALLBARS ) {
    return null;
    }
    
    if ( bInitialized == false ) {
        //WriteFileHeaders
        myFile = new File("LRtestFile.csv");
        sMkt = getSymbol() + "-" + getInterval() ;
        if (bTest == true) { 
            myFile.open("at+");     
            if(!myFile.isOpen()) { 
                debugPrintln("Could not open file!");
            } else {
            myFile.writeln("Mkt,Time,LR,LR1,LR2,LR3" );                        
            }
            myFile.close();
        } 
        bInitialized = true;
    }    
    
    if ( getBarState() == BARSTATE_NEWBAR ) {
    
    sum = 0;
    LR3 = LR2;
    LR2 = LR1;
    LR1 = LR; 
    
    Time = getYear(0) + "-" + getMonth(0) + "-" + getDay(0) + " " + getHour(0) + ":" + getMinute(0) ;        
    if (bTest == true) WriteFile();
    
    for(var i = Length; i > 0; i--) {
        sum += (i - (Length + 1) / 3) * close(i - Length);
    }    
    LR = 6 / (Length * (Length + 1)) * sum; 

    return LR; 
    }
}

function WriteFile() {
    myFile.open("at+");     
    if(!myFile.isOpen()) {
        debugPrintln("Could not open file!");
        } else {
        myFile.writeln(sMkt + "," + Time + "," + LR + "," + LR1 + "," + LR2 + "," + LR3 );
    }
    myFile.close();
} 

Here are the values:

Values on live recording
Mkt Time LR LR1 LR2 LR3
AB H7-5 2007-02-01 11:15 807.6406 807.6406 808.0557 808.4735
AB H7-5 2007-02-01 11:20 807.2062 807.2062 807.6406 808.0557
AB H7-5 2007-02-01 11:25 806.6378 806.6378 807.2062 807.6406
AB H7-5 2007-02-01 11:30 806.1338 806.1338 806.6378 807.2062
AB H7-5 2007-02-01 11:35 805.7154 805.7154 806.1338 806.6378

Values after data refresh
Mkt Time LR LR1 LR2 LR3
AB H7-5 2007-02-01 11:15 807.6538 807.6538 808.0698 808.4886
AB H7-5 2007-02-01 11:20 807.2185 807.2185 807.6538 808.0698
AB H7-5 2007-02-01 11:25 806.6492 806.6492 807.2185 807.6538
AB H7-5 2007-02-01 11:30 806.1443 806.1443 806.6492 807.2185
AB H7-5 2007-02-01 11:35 805.7249 805.7249 806.1443 806.6492

Thanks in advance,
Hamid.

[ACM]

Hamid
I ran your script in real time and the values appear to be the same as those returned on a reload.
Are you positive that you had setComputeOnClose() enabled when you ran your test in real time?
Alex

[hamidkassam]

Hi Alex,

I am certain I left the ComputeOnClose() both times - i.e. I did not change the efs between turning it up live from 11:15 to 11:35 and when I reloaded at 11:36. The steps I took in the test were as follows:

1. Load the script on chart
2. Wait for 25 minutes or 5 bars for output data to be written
3. Reload data (pressing Control Key and Left mouse click on OK in the data window
4. Stop script and examine the output file

I just reran the above procedure on GBP A0-FX on one-minute chart (from 17:56 to 18:01) and compared the results and yes, the values returned are same.

I wonder if this problem is related to specific source data servers or occurs only during the busy US market session? I will try it out again tomorrow on AB H7 to see what happens.

Regards,
Hamid.

[ACM]

Hamid
FWIW the reason I asked if setComputeOnClose() was enabled is because without that statement the efs will return incorrect values when used in real time. It will however return correct values on historical data because in that case it is executing only on completed bars, So you did not necessarily have to change the efs to get different values
Alex

[hamidkassam]

Alex,

I reran the script between 9:29 and 10:03 this morning and here are the results:

Real Time:

Mkt Time LR LR1 LR2 LR3
AB H7-5 2007-02-02 9:30 814.0113846 814.0113846 813.8683077 813.7901538
AB H7-5 2007-02-02 9:35 813.9363077 813.9363077 814.0113846 813.8683077
AB H7-5 2007-02-02 9:40 813.6772308 813.6772308 813.9363077 814.0113846
AB H7-5 2007-02-02 9:45 813.6203077 813.6203077 813.6772308 813.9363077
AB H7-5 2007-02-02 9:50 813.4350769 813.4350769 813.6203077 813.6772308
AB H7-5 2007-02-02 9:55 813.112 813.112 813.4350769 813.6203077

After data reload:
AB H7-5 2007-02-02 9:30 814.0113846 814.0113846 813.8683077 813.7901538
AB H7-5 2007-02-02 9:35 813.9363077 813.9363077 814.0113846 813.8683077
AB H7-5 2007-02-02 9:40 813.6772308 813.6772308 813.9363077 814.0113846
AB H7-5 2007-02-02 9:45 813.6052308 813.6052308 813.6772308 813.9363077
AB H7-5 2007-02-02 9:50 813.4209231 813.4209231 813.6052308 813.6772308
AB H7-5 2007-02-02 9:55 813.1138462 813.1138462 813.4209231 813.6052308

Notice how the values started changing from the 9:45 bar.

Just to clarify, when you say "if setComputeOnClose() was enabled" I assume that when that statement is in the efs I am essentially enabled? Or is there some other setting I have to use?

The above problem is a major concern to me because when I base a trading strategy on the values of LR1, LR2 and LR3 real time the results are different from when I apply the strategy on historical bars.

I would appreciate any further insight into this.

Thanks.
Hamid.

[hamidkassam]

urther to the post below, I have reformatted the values for easier reading in the attachment. Note the shaded values that changed.

Hamid.

[ACM]

Hamid
I ran the script in real time on a 1 minute chart of AB H7 from 10:45 to 11:30 at which point I made a copy of the LRtestFile and then refreshed the chart. The two files are contained in the attached zip folder (Copy of LRtestFile is the real time output and LRtestFile is after the refresh). As far as I can see the values are the same.
Alex

[hamidkassam]

Alex,

I also ran the script on one-minute AB H7 between 12:03 and 12:15. I attach the two files herewith. As you can see, the values were in sync from 12:03 until 12:11. From that time on, the values changed when data was refreshed.

From what I see in your zip folder the two files are one and the same. Try deleting the oriiginal file (after copying/renaming) before you refresh the data. I think you will get different results.

Hamid.

[eSignal_JasonK]

Hello Hamid,

I've been testing your code today as well and so far the data is matching on my end. I was also deleting the original file before executing the Ctrl-click refresh as you suggested.

I also used the Tools-->Data Export feature and compared the data before and after a refresh to eliminate any possible differences that may have been caused by the routines in the formula code. These data files are matching also.

Try this comparison on your end using the Data Export tool and see if you still get differences. If so, then the culprit may have something to do with the specific formula code. We'll continue to look into this next week as well.

[hamidkassam]

Hi Jason,

I re-tested between 3:49 and 4:11 PM on both the 1-minute and 5-minute charts. The values this time were identical on real time and after data refresh.

I also exported the data before and after refresh and the values in the output files were also identical.

I wonder if that's because of the time of day i.e. less busy etc. I know that this is a real problem for me as I have been experimenting using the LR values for certain signals for about a month now and have occasionally seen these signals shift (literally) on the chart 1 to 3 bars when I refresh the data. I had forwarded this to another trader and he noticed the same behavior.

After studying the problem I realized that the shifting signal was due to the LR values changing when data was refreshed. Hence the creation of test script to monitor the problem.

I will modify the script a little so as to show some sample signals on the chart for further testing next week. I will post the new efs file shortly.

Thanks for your input. I am confident there is a solution to this issue somewhere and with your and Alex's help I will find it.

Regards,
Hamid.

[hamidkassam]

Jason,

When using the Tools-->Data Export feature, does data not automatically refresh?

Hamid.

[ACM]

Hamid
I ran a test this morning from 9:15 EST to approximately 10:00 EST then after making a copy of the output file I deleted the original and refreshed the chart.
Other than the different values in the first 25-30 lines due to the fact that the efs does not reinitialize the global variables the data files appear to match across the whole period in which the efs was running in real time. The two data files are attached.
Alex

Originally posted by hamidkassam
Alex,

I also ran the script on one-minute AB H7 between 12:03 and 12:15. I attach the two files herewith. As you can see, the values were in sync from 12:03 until 12:11. From that time on, the values changed when data was refreshed.

From what I see in your zip folder the two files are one and the same. Try deleting the oriiginal file (after copying/renaming) before you refresh the data. I think you will get different results.

Hamid.

[hamidkassam]

Hi Alex,

Thank you very much for taking the time to further test this. I ran the tests several times this morning on the 1-minute chart also and the results in both files were also identical.

I can't explain this but I will continue to monitor the situation and will report any further problems with this.

Your attention is very much appreciated.

Hamid.

P.S. I am attaching a revised efs file - this has an additional parameter to change file name in between tests.

[hamidkassam]

Alex, Jason

I ran some more tests today on the AB H7 5-minute chart:

1. 9:35 to 10:30
2. 10:40 to 11:45
3. 11:50 to 14:00
4. 14:00 to 15:30

All times are EST. I found no discrepancies in data between real time and historical.

I will be out of town for the balance of this week so I will not monitor this for a while. But it does seem that the problems experienced last week and prior are gone. I can only conclude that they were due to bottleneck in data transmission either at my end or source servers.

Regards,
Hamid.