Chart from external file?

[senyorqueso]

Hi,

I'd like to chart data from an external file which would contain just one line with one number in it (the file would be continually updated by an external data source).

I'd like to read the file every tick and be able to produce a bar chart and be able to plot an MA of the chart.

The file i/o stuff is ok, but can anyone give me an idea of how to turn this single value into a plottable series?

Thanks,

SQ

[eSignal_JasonK]

Hello senyorqueso,

EFS does not have the ability to create a chart in the same style as a bar or candlestick chart that the Advanced chart displays. In other words, the chart data that the Advanced chart uses as its source to create its bar data can only come from the eSignal feed. However, you can use the Drawing Functions to simulate a candlestick appearance. See the eSignal_HeikinAshi.efs study in our library for a complete code example of how to use the drawing functions in this manner.

To get the data from your file (using File I/O) to create the OHLC data points for the current bar will require you to track the OHL values in some global variables. If possible, it may simplify things a bit if your data file would contain all four data points for the OHLC.

Once you've extracted the data from the file, you could return the variables that contain the OHLC data from the return statement in main() to plot them on the chart. They won't be in the form of a bar or candlestick chart. This is where you implement the technique from the Heikin Ashi study.

To calculate a moving average of the data, I have two ideas for you.

1)The old way would be to create a global Array object with a size equal to the length of the moving average period. At each BARSTATE_NEWBAR state you would perform a pop/unshift routine on the array to move the bar data back one element in the array to correspond with each completed bar. Once your array has data for each element in the specified size you would loop through it, sum up the data and divide by the number of periods. This result could also be plotted on the chart by returning it from the return statement in main().

2) The other idea isn't something I've tested so keep in mind this is just a theory. If you're able and willing to try this, it could be a much better option and would eliminate the routine of managing the array object by taking advantage of EFS2 functionality. Basically, the idea is to place the File I/O routine inside an efsInternal() call. This would be the only way to get the data into a Series object form that could then be passed to the sma() or ema() functions, which eliminates the need for the arrays managed by your formula code. The structure would be something along these lines.

var bInit = false;
var xData = null;

function_main() {
    if (bInit == false) {
        // create series object of file data
        xData = efsInternal("getFileData");
        bInit = true;
    }
    
    var nLast = xData.getValue(0);
    if (nLast == null) return;
    
    return nLast; // plots the data from the file
}

function getFileData() {
    
    // File I/O routine here
    
    return new Array(/*** your vars ***/);
} 

The other issue that you may need to deal with to make this work is the historical data of this file data. From what you've described, it sounds like only the current real time value is going to be present in the file excluding the history. You may need to add to the getFileData() function a routine that also puts the data imported from your real time file to a second file that will be used to store the history. In this thread you can find a detailed example of how this could be accomplished. In the getFileData() function you could also add a condition that checks the for the current bar index and extract the data from the history file to be returned from getFileData() until the formula is processing bar 0. At that point the getFileData() function would start looking to your real time file and returning the results from that file while simultaneously adding to the history file. The history file would only be used during the initial load of the formula of course so that the series object, xData, will have historical values. If this works, then the ema() or sma() function should be able to return valid results using xData as the source. The structure may be something along these lines.

var bInit = false;
var xData = null;
var xMA   = null;

function_main() {
    if (bInit == false) {
        // create series object of file data
        xData = efsInternal("getFileData");
        xMA   = ema(10, xData);
        bInit = true;
    }
    
    var nLast = xData.getValue(0);
    var nMA   = xMA.getValue(0);
    if (nLast == null || nMA == null) return;
    
    // plots the data from the file
    return new Array(nLast, nMA); 
}

function getFileData() {
    var nIndex = getCurrentBarIndex();
    
    if (nIndex < 0) {  // formula is loading
        // File I/O to extract data from history file
    } else if (nIndex == 0) { // formula is processing real time
        // File I/O routine to extract data from
        //      real time file
        
        // File I/O routine to append real time data
        //      to history file.
    }
    
    return new Array(/*** your vars ***/);
} 

Not a simple task you have in front of you, but hopefully this will give to some starting points to explore.

[senyorqueso]

Hi Jason,

Many thanks for your suggestions - I'll have a go at implementing option 2 first, and may have a go at adding the historical stuff after, but I'll be happy just to get it going in real time to start with. I think it should be less hassle to keep the external source as real time only - the source is a DDE feed into Excel and written out to a file on every change.

I expect I 'll have a question or two more for you soon...

Thanks,

SQ

[eSignal_JasonK]

You're most welcome.

[senyorqueso]

Memory problems

Hi Jason,

I've implemented this, without the efsinternal stuff, in the way it was done in the heiken ashi example, so I'm getting candle plots of the data from my external file just as I need. The file contains one line with 2 numbers, 1 of which gets plotted depending on the setting of a function parameter.

However, there seems to be a real problem with memory - the longer I let this run, the more memory gets used up, and eventually I have to quit eSignal and start again.

I've no idea what the cause of this is - do you have any idea?

Could it be the opening/closing of my external file on every tick? Maybe that ties up some memory which never gets released?

Is there a more efficient way I can do this?

The code is below - I'd be grateful for any suggestions you may have.

Thanks,

SQ

/*****************************************************************
File :        LR_xxx.efs
Provided By : LR
*****************************************************************/

function preMain() {

    setStudyTitle("LR xxx");

    setCursorLabelName("xxx", 0);
    setPlotType(PLOTTYPE_LINE, 0);
    setDefaultBarFgColor(Color.black, 0);
    setDefaultBarThickness(2, 0);
    debugClear();


    var fp0a = new FunctionParameter("Display", FunctionParameter.STRING);
        fp0a.setName("Display");
        fp0a.addOption("xxx");
        fp0a.addOption("yyy");
        fp0a.setDefault("xxx");
    var fp0 = new FunctionParameter("symbol", FunctionParameter.STRING);
        fp0.setName("Market");
        fp0.addOption("ES");
        fp0.addOption("NQ");
        fp0.addOption("YM");
        fp0.addOption("AX");
        fp0.addOption("ESTX50");
        fp0.setDefault("ES");

}


var O, H, L, C = 0.5;
var i = 0;
var index = 0;
var vBodyColor = Color.red;

function main(Display,symbol) {

  if (Display == "xxx") {
    setStudyMin(0.1);
    setStudyMax(0.9);
  } else if (Display == "yyy") {
    setStudyMin(0);
    setStudyMax(20000);
  }
if (getCurrentBarIndex() == 0) {

  fName = symbol + ".txt";

  f = new File (fName);

  if ( f.exists() ) {
          
    f.open("rt");
  
     while(!f.eof()) { 

      line = f.readln(); 

       if (line != null) {

         lineArray = line.split(",");
         xxx = lineArray[0];
         yyy = lineArray[1];
         xxx = OBD.replace(/\"/g,"");
       }
    }
    f.close(); 
  } else {
    debugPrintln("Can't find xxx File");
  }

  var nState = getBarState();

  if (nState == BARSTATE_NEWBAR) {
    // Rest Values to previous close
    O = C; 
    H = C;
    L = C;
    C = C;
    i++;
    //debugPrintln(i,"New Bar, i=");
  } else {

    if (Display == "xxx") {
      C = xxx;
    } else if (Display == "yyy") {
      C = yyy;
    }
    if (C>H) H=C;
    if (C<L) L=C;
    if (C>O) {
      vBodyColor = Color.green;
    } else {
      vBodyColor = Color.red; 
    }

  }
 

  drawLineRelative(0,H,0,L,PS_SOLID,1,Color.black,"Shadow"+i);
  drawLineRelative(0,O,0,C,PS_SOLID,3,vBodyColor,"Body"+i);
}
} 

[eSignal_JasonK]

Hello senyorqueso,

Your code is not limiting the number of drawn objects for your drawLine.... calls. You are assigning unique tag ID names correctly with the use of the i variable, however, as new bars are processed the collection of drawn lines is allowed to grow, which allows the consumption of memory to grow as well. In the eSignal_HeikinAshi.efs, the variable, iCntr, is reset to 0 after it gets incremented to a value above 200. This limits the collection of objects drawn on the chart to 200.

From eSignal_HeikinAshi.efs:

iCntr += 1;
if (iCntr > 200) iCntr = 0; 

Add the following line of code below your incrementing line for i.

i++;
if (i > 200) i = 0;  // add 

You can set the limit to a higher value if you'd like, but keep in mind that the larger the collection the greater amount of memory will be used as well.

[senyorqueso]

Hi Jason,

Thanks for your reply - I'll give i a go - but doesn't this mean that the lines drawn for the bars 200 bars back will disappear, as their tagID values are being re-used to draw the lines for the current bar?

SQ

[eSignal_JasonK]

Hello senyorqueso,

That is correct. This is how the eSignal_HeikinAshi.efs works as well.

[senyorqueso]

Still slooowww

Hi Jason - thanks.

I've modified this study and removed all of the line drawing stuff in an attempt to reduce the memory usage.

I'm now using setPriceBarColour to colour the current bar, the colour being determined by the value read in from the external file.

Unfortunately I still run into memory problems - the memory used by eSignal just keeps creeping up until eventually the whole thing grinds to a halt.

I'm using this efs on a 15s chart. Any ideas what might be the cause of this problem, or thoughts on how i can modify the code to make it more efficient?

Thanks,

SQ

/*****************************************************************
File :        LR_OBD_BARCOLOUR.efs
Provided By : LR
*****************************************************************/

function preMain() {

    setStudyTitle("LR_OBD_BARCOLOUR");
    setPriceStudy(true);
    setColorPriceBars(true);
    setDefaultPriceBarColor(Color.black);

    setCursorLabelName("OBD", 0);
    debugClear();

    var fp0 = new FunctionParameter("symbol", FunctionParameter.STRING);
        fp0.setName("Market");
        fp0.addOption("ES");
        fp0.addOption("NQ");
        fp0.addOption("YM");
        fp0.addOption("AX");
        fp0.addOption("ESTX50");
        fp0.setDefault("ES");

}


var O, H, L, C = 0;

function main(symbol) {

if (getCurrentBarIndex() == 0) {

  fName = symbol + ".txt";

  f = new File (fName);

  if ( f.exists() ) {
          
    f.open("rt");
  
     while(!f.eof()) { 

      line = f.readln(); 

       if (line != null) {

         lineArray = line.split(",");
         OBD = lineArray[0];
         OBD = OBD.replace(/\"/g,"");
       }
    }
    f.close(); 
  } else {
    debugPrintln("Can't find OBD File");
  }

  var nState = getBarState();

  if (nState == BARSTATE_NEWBAR) {
    // Reset Values to previous close
    O = C; 
    H = C;
    L = C;
    C = C;
  } 

  C = OBD*1;
  if (C>H) H=C;
  if (C<L) L=C;
 
  //debugPrintln(H,",H: ",L,"L: ");

  if (H>0.25) {
    setPriceBarColor(Color.red);
  } else if (H>0.2) {
    setPriceBarColor(Color.maroon);
  } else if (L<-0.25) {
    setPriceBarColor(Color.lime);
  } else if (L<-0.2) {
    setPriceBarColor(Color.green);
  } else if (((H+L)/2)>0) {
    setPriceBarColor(Color.purple);
  } else if (((H+L)/2)<0) {
    setPriceBarColor(Color.olive);
  }
}
return;
} 

[eSignal_JasonK]

Hello senyorqueso,

Can you post a copy of your OBD file so that I can test it?

Also, you may want to declare your OBD variable in the global scope. The formula generates a syntax error on, C = OBD*1; because OBD only gets created if the OBD file is present.

[senyorqueso]

Jason,

Thanks for looking into this - the file format is very simple - a one line file containing 2 numbers as follows...

"0.16",1921

The efs i posted only uses the first number.

If you create a file called ES.txt and paste the above line into it it should work fine. In my case the file gets overwritten continuously by Excel, but I don't think this should make a difference in performance.

Thanks,

SQ

[eSignal_JasonK]

Hello senyorqueso,

I've been testing this scenario for most of the day on both 8.0 and 10.0. So far, I'm not seeing any problems. The code appears to be working efficiently. At this point I do not think the source of your memory issue is related to the formula code.

The only thing I can think of currently is that your tick chart that is collecting the tick data as time moves forward could be the problem. Do you restart your application each day or are you allowing it to run continuously for a number of days?

[senyorqueso]

Hi Jason,

Thanks for trying it out. I've actually seen better performance over the last few days after cleaning up my pc and increasing the page file size, although it does still seem to slow down the application after running for a day. Without this efs running i don't see this.

It's certainly usable now, but something still seems not quite right.

Anyway, I can carry on as I am with a daily re-boot, so no need for you to waste any more time on it.

Thanks,

SQ