setGlobalValue()

[clearpicks]

Please check the implementation of setGlobalValue(). I find out it consumes too much CPU time. Because Esignal doesn't allow more than 3 symbols in each efs, I have to use setGlobalValue() and getGlobalValue() to distribute some computation to multiple scripts, the efficiency of setGlobalValue() seems a non-negligible problem here. You may simple apply Other\GlobalValueTest.efs and watch its execution time in EFS Performance Monitor.
It takes average 19.06ms for each main() call.

[eSignal_JasonK]

Hello clearpicks,

I've tested the GlobalValueTest.efs on a 1 min chart of IBM with no other formulas running and I'm getting an average time of around 0.67 ms. Please tell me more about your specific setup so that I can recreate the problem.

Thanks,

[clearpicks]

You can try this scenario

Hi Jason,

I can not post my scripts and settings. However I created this scenario which should recreate the problem I am talking about.

N indentical advanced charts, each showing $SPX 1min chart with Other\GlobalValueTest.efs, using the default dynamic time template (0:00-24:00)


N Total calls Total Time Avg Time
1 792 282 0.3561
5 3960 5359 1.353
10 7920 20391 2.575
15 11865 46549 3.923
20 15820 73700 4.659
25 18216 99591 5.467
30 23760 165847 6.98

From the above test results, we can observe that something is obviously wrong. Even with one chart running, the 0.35ms average execution time is still questionable comparing the execution time of other complicated efs which don't call setGloblaValue().

In my trding layout, the average time for setGlobalValue() execution is even worse ( arround 22ms ~ 60 ms depends on how many charts and efs I was using ).

I guess the problem comes from the implementation of setGlobalValue(). Considering the speed of other api's, this is really a potential big performance bottleneck. Right now I have to carefully code my efs to remove all unnecessary calling of setGlobalValue(). However, for every 1 second realtime, my CPU is still busy for 1/3 second. Most of them are used by setGlobalValue().

[clearpicks]

Jason,

Have you recreated the scenario?


Clearpicks

[eSignal_JasonK]

Hello Clearpicks,

I have tested your scenario and experienced very similar results. Using setGlobalValue() by itself is a heavier call than getValue(), for example. In your particular scenario, I think we are seeing an increase in the Average Times due to the sheer number of charts using setGlobalValue() plus the fact that they are all affecting the same global variable. I think the results we are getting with this scenario should be expected.

Perhaps we could take another approach to this problem. Please describe for me what data you need to share among your various formulas in your layout. Post some of your code that revolves around your setGlobalValue() calls. Maybe we can create a more efficient solution for your specific formulas.

Regards,

[clearpicks]

I don't think the significant increasing of avg. timeit is due to the fact that all charts are accessing the same global value. You can specify different global variable name to set in my testing scenario for each chart. In my trading system, i have the following
script running on ($TRIN, T) and the avg. running time is 32ms.
It is the only script which set the global variable "Gtrin" and "Gtrin1" on my screen. Also, during trading hour, if I comment out the "setGlobalValue()" in any specific script, in the performance monitor window, the avg. time of that script always goes down dramaticly. You need to reset the performance monitor after the study is brought up, then enable the monitor again to monitor the execution time of the current bar.

I suspect the overhead comes from the thread syncronization
(the study thread and some thread maintaining global variables). I don't know how esignal implement setGlobalValue(), therefore it is only my wild guess. Anyway considering the lightening fast of all other studies, the overhead of setGlobalValue() seems very annoying.


/*
* script running on ($TRIN, T) to set current $TRIN and
* yesterday's $TRIN to some global variables.
*/

function preMain()
{
setPriceStudy(true);
}

function main() {
if ( getCurrentBarIndex() == 0 ) {
var trin_yesterday;

trin_yesterday = getValueAbsolute("Close", -1, "$TRIN,D");
setGlobalValue("Gtrin", close());
setGlobalValue("Gtrin1", trin_yesterday );


}
}

[eSignal_JasonK]

Hello Clearpicks,

I think using setGlobalValue() in your case in unnecessary. Setting Gtrin1 to yesterday's close only needs to be done once. Yesterday's close can't change so calling setGlobalValue("Gtrin1", trin_yesterday) on every tick is creating a great deal of unnecessary overhead. If you have other formulas that need the same Gtrin and Gtrin1 data, you should use getValueAbsolute() and close() within those formulas as I have in the example below. Set that information to a local variables that are var'd globally within your formula. This will solve your performance problem.

The following example gives you the same results with an average time of 0.022ms:

function preMain() {
    setPriceStudy(true);
}

var Gtrin = null;
var Gtrin1 = null;

function main() {
    if (Gtrin1 == null) {
        Gtrin1 = getValueAbsolute("Close", -1, "$TRIN,D");
    }
    
    if (getCurrentBarIndex() == 0) {
        Gtrin = close();
        return new Array(Gtrin, Gtrin1);
    }

    return;
} 

[clearpicks]

My trading system is much much complex than the script which set $TRIN to a global value. Basicly it involves the cooperation of more than three dozen scripts. Thanks to the 3 symbol limitation imposed by Esignal, which really made my programming not that straightforward and overly complicated. Actually I can not find any reason to justidy the 3 symbol limitation, because if the purpose is to prevent people from implemeting their own scan program ( so that Esignal can sell its own market scanner addon program ), then the user can still use the resetEFS() to bypass such hurdle. If the purpose is solely to limit the number of symbols accessible in an EFS so that "to prevent the Esignal server being overloaded", then a use can still open a lot of windows or simply to apply a lot of different EFS' or even the same EFS a lot of times within the same advanced chart to easily "overload" the server. In this scenario, the 3 symbol per EFS limitation can never effectively prevent a malicious user from doing any possible bad thing. Actually the 500 hundred symbol limitation has already been imposed by Esignal, which is really the true doorkeeper.


Clearpicks

[eSignal_JasonK]

Hello Clearpicks,

It is my understanding that the 3-symbol limit was established to prevent a single formula from running into cpu issues. EFS was not intended to be a market-scanning tool. EFS is a powerful tool but it does come with limitations, which your development efforts here seem to have exceeded. Your attempt to develop an overlay scenario to exceed the 3-symbol limit is very creative, but as you have discovered, it leads to a cpu problem. I’m not entirely sure that there is a solution for the level of complexity of your overlay strategy. At this point, I would recommend contacting one of our EFS Development Partners. They may be able to come up with some other advanced solutions.

Best Regards,

[clearpicks]

Hi JasonK,


According to my observation and experience, for an EFS of reasonable complexity, the CPU issue is never a big issue. For today's computers, those line drawing and methimatical computation are only a piece of cake. Those trouble makers I encountered are mediocre thread scheduling and syncronization algorithms, bursty tick data coming from the server. The 3-symbol limit really can not do any good to prevent an over-complicated EFS making the CPU freezen even if it only uses one symbol. On the other hand, a very complicated EFS which accesses even 500 symbol can still be well designed and coded so that only a small fraction of CPU time is used ( for example, I can add timers in my EFS so that it only calculates every a few seconds or every fixed number of ticks, or I distribute my calculation into several steps and alternatively execute one step per tick data to relieve the CPU burden). There are a lot of tricks to prevent the CPU issue and I believe it is solely the user's responsibility to limit the number of symbols being used in a single EFS if the local CPU usage becomes a issue. It is also the Esignal subscriber's right to use whatever number of symbols in an EFS as long as the total number of symbols don't exceed 500, which is limited by the subscription agreement. I agree every trading software has its limitation, but I just see no point to set the 3-symbol limit ( the reason I have explained before ). To be a truelly competitive trading application, Esignal should give the most flexibility and pass the whole power to its users without setting such kind of strange and unreasonable limit as long as it don't have side effect on Esignal's data server and the speed of other users. Do I miss some other reason which can justify the 3-symbol limit?


Clearpicks

[gspiker]

clearpicks,

I agree with you that 3 symbol limit is silly, but I disagree that there are no issues about CPU issues wrt this. Clearly you increase the probabilty that users will do something silly and clog their puter, or put extra drain on the server. However, there are other ways eSIgnal could limit this if they choose to. Limiting the symbol count per EFS isn't optimal.

Garth

[clearpicks]

Hi Garth,


I agree the more symbols are accessed, the more CPU time will be consumed because for each additional symbol, at least one getValue() or similar API will be called. However that kind of overhead is much less significant as setGlobalValue(), for example, the average execution time of

Gtrin1 = getValueAbsolute("Close", -1, "$TRIN,D");

is only 0.022ms compared with several milliseconds even tens milliseconds if setGlobalValue() is used in an EFS. The reason I used setGlobalValue() and getGlovalValue() in my previous example is solely because one of my EFS need to access more than three market statistics although the computation itself is pretty simple. Moreover the users obviously should understand each line of the EFS code and the computational complexity of the EFS. I don't know how a people can trust an indicator or an EFS without fully understanding how it is constructed or coded. Therefore Esignal only need to warn the users of the potential performance deterioration and these is really no reason to set the 3-symbol limit.

Regards,
Clearpicks