Questions about data series in external intervals

[mbuta]

EFS2 allows us to call values of some data series in external intervals, WITHOUT having to declare those data series [as we have to do with studies].

For example, we can use
var Myvar1 = open(-1,inv(XX));
anywhere, anytime, in any base interval.

I have the following questions:

1. Is EFS2 engine "compiling" these specific values as we request them, or it generates the entire data series when first encounters that call?
I am particularly interested when XX is an interval smaller than the chart interval.

2. If the entire data series is created, and XX is an interval smaller than base [chart] interval, for how many bars?

3. Are there any "default" data series for external intervals that EFS engine generates for a symbol, regardless of the base interval?
If yes, for which intervals and what data series?

4. Assuming that same data series and/or study [with exact same parameters, in diferent panes] is invoked/used, is EFS engine duplicating the study/data series for each pane?

5. We know that "reloadefs()" preserves global vars. Studies are also global vars [must be declared as such].
If reloadefs() is used to load and show diferent configuration [diferent studies, plots], what hapens with previous studies [if I do not reset them in purpose on reload efs]?

I am asking all these questions for two reasons:
a/ I want to minimize the EFS engine overhead and
b/ Apprently, lower time interval data series create abnormally high overhead, compared with all others.
I tried several ways to detect the "1S" boundary and keep running into all sorts of problems and troubles.

Thank you

[Doji3333]

Re: Questions about data series in external intervals

First off, I don't know the exact answers to many of these questions as they are really directed towards esignal programmers to answer, but I will try to offer advice.

1. Is EFS2 engine "compiling" these specific values as we request them, or it generates the entire data series when first encounters that call?
I am particularly interested when XX is an interval smaller than the chart interval.

If you're calling "re-initializing" the value every time before you call it, it could be creating some overhead issues. I've found I have to create "instances" of certain functions and requirements with my systems.

I'm not exactly sure about this, I'm sure Alex or someone from esignal will reply with the correct answer, but as I understand it, a data series is created and maintained in RT at the time you call it. So if you are calling these over and over within your code, you can first try to reduce the overhead/load by reducing the call to ONCE in your code, then compare variables. Like this..

var vtempInvDataArray = new Array ( open(-1,inv(XX)), high(-1,inv(XX)), low(-1,inv(XX)), close(-1,inv(XX)), vol(-1,inv(XX)) );

if (vtempIncDataArray[0] < vtempIncDataArray[3]) {
  //  Black candle

}
if (vtempIncDataArray[0] > vtempIncDataArray[3]) {
  //  White candle

} 

Each subsequent (sp?) call compares local variables instead of making another call to the external data.

2. If the entire data series is created, and XX is an interval smaller than base [chart] interval, for how many bars?

I would assume it uses the time template you are running on yoru chart. I've started using 3+ times the minimum number of bars for your system - like 300~500 bars as a time template.

You might try this to see if it helps?

3. Are there any "default" data series for external intervals that EFS engine generates for a symbol, regardless of the base interval?
If yes, for which intervals and what data series?

I don't believe so. I've found I call what I need in the code (at the end of a bar of in RT - as it ticks) and each call accesses the "esignal data manager servers" in RT.

4. Assuming that same data series and/or study [with exact same parameters, in diferent panes] is invoked/used, is EFS engine duplicating the study/data series for each pane?

Yes. I don't believe there are any optimizations of this nature in esignal. I believe these act as unique components of a chart - duplicates.

5. We know that "reloadefs()" preserves global vars. Studies are also global vars [must be declared as such].
If reloadefs() is used to load and show diferent configuration [diferent studies, plots], what hapens with previous studies [if I do not reset them in purpose on reload efs]?

Good question?? I would imagine they are left functional (in memory) unless there is a "teardown" procdedure that destroys all external data series objects from the efs scripting engine parser.

I am asking all these questions for two reasons:
a/ I want to minimize the EFS engine overhead and
b/ Apprently, lower time interval data series create abnormally high overhead, compared with all others.
I tried several ways to detect the "1S" boundary and keep running into all sorts of problems and troubles.

Thank you

I know there are some issues with running larger, more complicated efs scripts, custom intervals and overhead. With my scripts, it's best to "reboot my pc" every Sunday night and try to let esignal run all week. Short term TICK or VOL charts will quickly eat up memory as your system runs all week and the chart collects data. That's why I use 300+ bar time templates to help on an "esignal reload" or for my systems to run.

Another way to do all this (and forget all the overhead) is open another chart (the lower interval) and create a "feeder" system from the second chart. It will cut down your overhead substantially because you'll only get the data ONCE for historical and as needed for RT work.

You simply create another EFS file that dumps the data into a "GlobalValue" as an array (each bar's data). Create two arrays and collect all the data you need and dump that array into Global Memory (with setGlobalValue() ). The first one is a historical record of all data (if needed) that can be used at any time. The second one for use in RT operations (and much smaller - like 10 bars). Set the first only to update on new bars and the second needs to update in RT (updating the GlobalValue arrays).

Now, instead of making multiple calls to the DataManager every bar of tick, you would call the single "getGlobalValue" (as needed) on each bar or tick. Use the historical array for non-rt operation (load it once, and only once, with an "init" function), then when you get to the current bar, start using the second array.

I'll bet with the use of two charts with setup time-templates and the feeder method, you could get this EFS humming along all week.

[Doji3333]

heck, you could even try to destroy the historical data on the other chart by passing a third global value back to the lower interval chart and recalling the variable as NULL.

[mbuta]

Hi Brad and thank you for sharing your experience.

As you said, let's hope that eSignal people will come with some definite answers.

As per your comments:
a/ I also run 600 bars templates, to get minimum 300 useful bars. CCI needs about 3x its period for first value.

b/ The only reason to use "1S" series is to detect busy times in RT, and that is done only in one place and only once.

c/ I tried using GlobalVars, but overhead seems to be greater than creating the "1S" info in each bar [specially if it is already in eSginal server and don't need to create my own data series].
At that time, I create the data series in the price pane and accessed it from non-price panes.
There is one more [serious] issue with using GlobalVars: EFS engine does not guarrantee an order of execution of diferent panes and cannot force one. Therefor, one never knows if the GlabalVar was or not updated for that particular tick [unless more complex logic is implemented, which defeats the purpose of being quick and efficient in busy times].

d/ I also believe that all called studies keep running in background [have many "signs" of that], just need a confirmation.

e/ I cannot afford to keep the eGinal running all week. I have to re-start it couple of times a day. The reason, I trade options and "navigate" through many, many symbols during the session.
I increased the heap and stack size to 132Mb and 512Kb and that is a lot of room.

Good to hear from you. How did the trip to eSignal go?

[Doji3333]

It seems to me that if you are running a "1S" dataseries to determine busy times in the market, could you not simply calculate this information in RT within your script using the tick data and the time functions?? I don't know if this will help, but given the fact that while you are running this in RT, it could essentially be building/calcing your "busy indicator" from internal data it has collected??

I also understand the issue of GlobalValues, but in my opinion a "feeder/reader" type of solution would only result in a 1 tick delay (in most cases) and that is something I can live with.

You might try loading up the windows task manager and track the memory usage of esignal as you change symbols and other actions. You might find it is hogging memory somehow??

[mbuta]

Brad,

Theoretically, you are correct.
In reality, there is no "cheap" way [in terms of overhead] to get the time intra-bar.

I still hope eSignal people will answer my questions. I hope.

[ACM]

Re: Questions about data series in external intervals

Mihai
Adding to the replies that Brad already provided.

...WITHOUT having to declare those data series [as we have to do with studies].

Actually you can do the same with the builtin efs2 studies ie call them without declaring the series eg sma(10, -1) will retrieve the prior value of a 10 period simple moving average. This is explained with detailed examples in this thread.

1. Is EFS2 engine "compiling" these specific values as we request them, or it generates the entire data series when first encounters that call?

The entire series is created the first time the value is retrieved

2. If the entire data series is created, and XX is an interval smaller than base [chart] interval, for how many bars?

Depends on the Time Template used for the external interval and/or if setIntervalsBackfill() is enabled or not

. Are there any "default" data series for external intervals that EFS engine generates for a symbol, regardless of the base interval?
If yes, for which intervals and what data series?

No

4. Assuming that same data series and/or study [with exact same parameters, in diferent panes] is invoked/used, is EFS engine duplicating the study/data series for each pane?

The data is not duplicated. When calling an external symbol/interval it is the same as creating a chart for that symbol/interval and this is referenced by all scripts based on that data. Each study though runs its own instance of the series object.

5. We know that "reloadefs()" preserves global vars. Studies are also global vars [must be declared as such].
If reloadefs() is used to load and show diferent configuration [diferent studies, plots], what hapens with previous studies [if I do not reset them in purpose on reload efs]?

Studies don't have to declared as global variables. While it is more efficient to do so they can be declared as local variables.
Series objects are maintained for the life of the efs.
Alex

Originally posted by mbuta
EFS2 allows us to call values of some data series in external intervals, WITHOUT having to declare those data series [as we have to do with studies].

For example, we can use
var Myvar1 = open(-1,inv(XX));
anywhere, anytime, in any base interval.

I have the following questions:

1. Is EFS2 engine "compiling" these specific values as we request them, or it generates the entire data series when first encounters that call?
I am particularly interested when XX is an interval smaller than the chart interval.

2. If the entire data series is created, and XX is an interval smaller than base [chart] interval, for how many bars?

3. Are there any "default" data series for external intervals that EFS engine generates for a symbol, regardless of the base interval?
If yes, for which intervals and what data series?

4. Assuming that same data series and/or study [with exact same parameters, in diferent panes] is invoked/used, is EFS engine duplicating the study/data series for each pane?

5. We know that "reloadefs()" preserves global vars. Studies are also global vars [must be declared as such].
If reloadefs() is used to load and show diferent configuration [diferent studies, plots], what hapens with previous studies [if I do not reset them in purpose on reload efs]?

I am asking all these questions for two reasons:
a/ I want to minimize the EFS engine overhead and
b/ Apprently, lower time interval data series create abnormally high overhead, compared with all others.
I tried several ways to detect the "1S" boundary and keep running into all sorts of problems and troubles.

Thank you

[mbuta]

Alex thank you.

Let me summarize what I understood. If not correct please let me know:

1. There is no [time/overhead] advantage in just calling a data series value, because the EFS engine does not have any "default" data series already generated.
Might as well create it when most convenient.

2. Lower interval data series are generated only for the same number of bars as the base interval [from Time Tem plate].

3. If the same study, with the same parameters, for the same symbol and interval [base or external] is used in several panes/charts EFS engine keeps only one copy of it.

Note: do not see the advantage of a study as local var, but it might be one.

Thank you.

[mbuta]

Alex,

FYI: Statement #3 is incorrect. EFS does repeat every study in every pane it is called.

Also, I hope the assertion that EFS engine generates [and keeps running] the entire data series is incorrect. It would be a very poor design to create such an overhead, only because I call one open/close value once and never need that again.

Too bad that eSignal does not care to respond to these type of questions. Real bad. I have other two posts that nobody bothered to respond to for a week now. They all are specific questions about EFS engine internals.

Very dissapointing.

[ACM]

Mihai

FYI: Statement #3 is incorrect. EFS does repeat every study in every pane it is called.

I know that. In fact in my previous reply I stated that while the underlying data is common to all the studies each study runs its own instance of the series object
With regards to the series objects these are maintained for the life of the efs as I indicated earlier and as stated also elsewhere in this forum.

2. Lower interval data series are generated only for the same number of bars as the base interval [from Time Tem plate].

The number of bars loaded for an external interval [assuming setIntervalsBackfill() is not being used] depends on the definition set in the Time Template for that interval. If for example in the Time Template you have the 2 minute interval set to load 200 bars and the 1 minute interval set to load 100 bars then calling the 1 minute external interval from the 2 minute chart will load 100 bars of the 1 minute interval.
This is clearly illustrated in the screenshot enclosed below where I am running the following script

function main(){
    return (close(0, inv(1))
} 

in both charts.
If there is no definition for a specific interval then the bars will be loaded according to the Time Template's default settings
Alex

Originally posted by mbuta
Alex,

FYI: Statement #3 is incorrect. EFS does repeat every study in every pane it is called.

Also, I hope the assertion that EFS engine generates [and keeps running] the entire data series is incorrect. It would be a very poor design to create such an overhead, only because I call one open/close value once and never need that again.

Too bad that eSignal does not care to respond to these type of questions. Real bad. I have other two posts that nobody bothered to respond to for a week now. They all are specific questions about EFS engine internals.

Very dissapointing.

Originally posted by mbuta
Alex thank you.

Let me summarize what I understood. If not correct please let me know:

1. There is no [time/overhead] advantage in just calling a data series value, because the EFS engine does not have any "default" data series already generated.
Might as well create it when most convenient.

2. Lower interval data series are generated only for the same number of bars as the base interval [from Time Tem plate].

3. If the same study, with the same parameters, for the same symbol and interval [base or external] is used in several panes/charts EFS engine keeps only one copy of it.

Note: do not see the advantage of a study as local var, but it might be one.

Thank you.