it seems it takes forever to load an 10second chart

[clearpicks]

I use 10second as the time interval on some charts. To optimize some parameters, I need to export the historical 10sec data from charts. The miximal data for 10sec data (which is built from tick data) is 10 days. So I specify a timeframe of 10S for 10Days. However it seems it takes forever to load a chart with such time frame. For example, it just took a few minutes to finish the loading of a YM M2008, 10S ( 10 days ) chart. In the cursor window, it was shown it first stoped at barcount = 51, then it kept showing "Receiving..." but the barcount did not change at all. After a few minute, it changed the barcount to 11xxx, then after several minutes more, it changed to 22482 and "receiving..." was changed to "OK". It could never finish the loading of ES M2008. I only opened one chart without any other symbols loaded or any studies applied. I understood the downloading of tick data takes quite some time, but it is definitely something wrong here. IT SHOULD NEVER TAKE SO MUCH TIME TO LOAD SUCH A CHART. It seems there must be something wrong in the implementation of esignal client or the data manager.

I did the test on esignal 10.0 and 10.1 beta (1184) and there were no much difference on these two versions.

- Clearpicks

[clearpicks]

If I ctrl-click on "Receiving..." to reload the chart, the chart may just become blank and barcount is reset to 0 and is stuck in that state forever.

At the first loading of such a 10 second chart, the CPU usage shot to above 90%. Then it droped down to some low level.

When I am writing this post, the loading of ES M2008 is finally finished because I saw the CPU usage shot to about 50% for a few seconds. Then I switched to the eSignal screen and as I expected, the ES M2008 chart was just been loaded and showed 59954 bars on it. The CPU usage droped to 1% thereafter.

How does esignal send historical data request and how does esignal charting engine draw charts? In old esignal version (7.x?) it is very slow when loading intraday chart such as 60min charts because each time it only request one day's data. So for 60min chart, each time it only request 8 bars (depends on the start and end time). How does esignal request such 10 day tick data? Does it request the tick data in one request or cut the 10 days into several small interval and each time only request tick data for one interval? How does esignal set its timeout timer so that it may resend the request if the request gets lost or incoming data packets get corrupted/lost? What is the timer backoff algorithm and flow control algorithm?

- Clearpicks

[clearpicks]

Did eSignal make any change in data manager and server side regarding how to handle the downloading request of tick data? Back in september last year I used 10 days 10second charts a lot and I don't think I noticed such slow loading issue at that time.

- Clearpicks

[clearpicks]

Definitely this is an implementation issue instead of a network issue. For all symbols I tested (all were those actively traded futures contracts such as YM, ES, 6E, 6J, ZG, etc.) esignal behaved in a very similar pattern. It first showed about 60 bars (10 minutes?) and stop showing any bars untill all bars were downloaded. During that downloading period (sometimes it takes more than 5~10 minutes), I saw no activity on esignal screen.

[ACM]

clearpicks
I believe eSignal is already aware of the issues with tick-based intervals and from what I have seen is already working on resolving them.
Alex

[clearpicks]

I am not sure whether esignal has been aware of this or not. Because it has taken so much time to load the historical tick data, when all bars are finally downloaded, the chart can not be updated in realtime anymore. For example, I opened a CL M8 10second chart at 12:21:11, at 12:36:xx when it was finally finished, the last bar on chart was still with timestamp 12:21:11 and it never updates thereafter. Obviously the realtime streaming session got closed during that 15 minute downloading time..


- Clearpicks

[ACM]

clearpicks
This has been suggested before (see this thread) and as far as I can see eSignal has already indicated that it will take it into consideration.
Alex

Originally posted by clearpicks
Alex,

Thank you for your reply. I think esignal should have a webpage for bug tracking and reporting purpose so that the users would be able to know which bugs have been reported and confirmed and which bugs are being worked on and fixed, so that it would not only save esignal technical support team's time but also more improtantly the users' time. For example, for the set bar background bug in 10.1 beta alone I saw quite a few threads reporting that same issue. Also regarding the drawText flashing issue, although I reported right after esignal 10.0 was out, when 10.1 beta was out, I found it was not still not fixed yet. If there were a centralized webpage for both esignal team and the users interactively managing/report/tracking those issues, all these would not happen.


- Clearpicks

[clearpicks]

I downgraded esignal from 10.1 beta to 8.0 and it is as slow as 10.1 when opening 10sec charts. So I think the problem must be on the esignal server side.

[ScottJ]

We have developed and released a new tick server that is capable of handling much "smarter" request types so we can speed up chart retrieval. Right now, an entire day's worth of ticks are requested so for active issues like the emini's, we're attempting to send a tremendous amount of data down to you.

Once 10.1 is completed, you should see some improvements as the desktop will start making those "smarter" requests and we'll continue that improvement in 10.2 and 10.3.

Thx

Originally posted by clearpicks
I am not sure whether esignal has been aware of this or not. Because it has taken so much time to load the historical tick data, when all bars are finally downloaded, the chart can not be updated in realtime anymore. For example, I opened a CL M8 10second chart at 12:21:11, at 12:36:xx when it was finally finished, the last bar on chart was still with timestamp 12:21:11 and it never updates thereafter. Obviously the realtime streaming session got closed during that 15 minute downloading time..


- Clearpicks

[ScottJ]

Also, when you do a T&S request, do you see historical bid/ask data? If so, you are connecting to a server that is storing up to 70% more data than a server that does NOT store b/a data. If you don't need historical b/a data, we'd recommend you using the new lighter servers. Contact TS to change your routing.

Thanks.

[ScottJ]

Me again. I checked your account and it appears you are already going to our lighter load servers.

On a positive note, we just released another beta internally and the difference (vs prior versions) in loading intraday and tick charts is unreal. We hope to post a new beta any day now. Keep an eye on this thread for a new release.

10.1 gold is going to make a lot of our users very happy....

Thanks.

[Dave180]

clearpicks
Bit late here, but in case its useful, try using the Tools / Tick Replay / Downoad
that way you can both "prime" eSignal with the tck data (do it before you open the chart), and get a continuous count of the tick data download.
Also, a Standard Chart set to Tick is a way of checking what is in the eSig "cache".

[clearpicks]

Dave,

Thank you for your tip.

The point of my posts (some of them in another thread) is that the design of eSignal regrading to the loading second charts is flawed. For those actively traded instrument, the number of bars for each day is much much smaller than the number of ticks. However in eSignal current design, all tick data is pushed to the user side and sub-minute OHLC series are built from raw tick data. If that series can be built on the server side per request, the data transfer would be much less and loading time of sub-minute chart would be much faster. I understand storing second data series might consume extra storage capacity. However it is only a little fraction of that of tick data for those actively traded instrumens.

Another possible solution to speed up the loading of sub-minute charts is to allow the user to specify the number of bars to load when the time frame is "S". For example, if I specify 180 bars for "10S" time frame, and eSigmal still "smartly" and incrementally requests tick data (i.e. 500 ticks for each request) and auto stops after enough ticks have been received to build the last 180 bars, this should significantly speed up the loading time. For example, in case of "ES M8, 10S", it should take about one tenth of that in the current design in which I have to download 24 hours of tick data. (If I set it to dynamic, any scroll of chart to the right may cause the script to be reloaded because more data are downloaded, which is not desirable in some case when you want to use EFS to do some auto trade).

- Clearpicks