Yet another data (es tick) and or repaint / display problem

[Dave180]

"A diary of a few wasted hours in the life of an eSignal user" (actually "to use" is the opposite sense here, but you get the idea)

Hi
OK, someone explain this set of observations, I am sick to death of it all:
Yesterday, to a page of various non-tick (1 minute mainly) charts, add an ES #F 20000V chart (has custom EFS, but often often used before).
Find that the winsig goes to maxed-out core, seems to take about 5 minutes to repaint. I don't think this is a data problem as I left it running all day (every few minutes the charts would flicker into view, but then try to repaint themselves) and then discovered that when the market closed the repaints were still taking as long. In fact even with the DM closed it STILL took several minutes to repaint the 20000V chart.
Change the 20000V chart to 5min. All problems go away(do this by waitiing fir a repaint to finish and entering 5 and enter). Change back to 20000V see problem come back.
Delete all indicators. (when in 5 min mode), change back to 20000V, same 5min repaint problem.
New page, 20000V chart with no indicators only, same slow repaint 100% core problem.
So what's that then?


With new DM instance (see netstat below)

Not having written notes at the time, I can't say for sure what the circumstances were for this next subsequence, but basically I did a tick download of es #f 10days and it seemed to finish almost instantaneously. After the download I displayed a Standard tick chart, it looked fine with 10days of data. I then opened an es #f 5min chart, fine again, but when I changed it to 20000V instead of the chart appearing instantly (using data already downloaded) it seemed to start downloading the data again, and failed. The cursor flashed up "no TS" for a moment then went to OK and the current bar seemed to update, but no historic bars were displayed. Tried to do a tick download of 10days, absolutely nothing happened (no download).

Now back to better documented stuff...
Now open eSig with blank page
Open Standard chart, set to es #f Tick, see 1 day of data appear quite quickly.
Change to 10 days, watch as the tick chart painfully redraws continuously as the data grows backwards in time (I've mentioned this before), eSignal is unresponsive during this, takes about 15minutes to get to midway through 19th, then no more data seems to arrive, but the chat continually redraws.
And while it is drawing notice that of the three lines it draws (green, light blue, dark blue) the green and dark blue lines have a flat-line area from about 1/4 of the way through 21st to 1/4 way through 22nd. Kill eSignal.

Restart eSig, tick download 10days ES #f, see from rolling status it downloads the data on the 19th without problem, and continued with 16th, 15th, took the normal few minutes to complete. Display a Standard 10day tick, see it appear almost instantly with data going back to late on 14th (the 14th is the first day on the chart). Then it sits there redrawing itself (100% core again) and is unresponsive. The discontinuity seen on 21st to 22nd is not there, but there do seem to be an odd looking very short flat line in the last 1/4 of 27th (in all colours). Give up waiting after a few minutes, kill eSig.

new esig, adv chart, no indicators, es #f 5mins (10days 24hrs) seems fine, data back to start of 15th. Change template to 3 days (to avoid the latest hiccup over the problem with more than a few days of tick data sometimes being a problem I've read about). Change to 20000V, data downloads in a couple of minutes or so and chart displays with no problems, starting at 16th (that is more than 3days, more like 7 and a bit). Change template to 4days, flip to 5mins and back again, see still starts on 16th, with 5 days, still starts on 16th. Change to 8 days, still starts on 16th. Change to 10 days, still starts on 16th. But at least it is being displayed. With this chart open, open my "normal" 20000V chart (the one that had problems with the repaint). eSignal freezes (appears to be a repaint problem again, but this time it won't even stabilise enough to try changing the 20000V charts to, say, 5mins, kill eSig.

New eSignal, download 10days es #f, takes the normal few minutes (goes back to start of 15th).
With an $SPX 4 min chart and a 3 day template why do I only see bars for the 27th? Just a minor point.
Change to es #f 20000v (3day template). One thing I notice is tha tit says "Dynamic" not "3 days" as it did for the $SPX chart? Why. And I have data back to 15th being displayed.
Close chart. Open my normal "20000V" chart, but a version saved as 5min 1000Bars, it opens fine.
Change to 20000V, still 1000Bars, eSig goes into max cpu but it does display it after a minute, again I notice that it says "Dynamic", not 1000bars! However it is now in the "slow repaint" mode again. Can't get to change back to N mins (oddly, I can't even change back to "n mins" as every time I click on the chart to give it focus it seems to go into a redraw).

Meanwhile, I can open a 200V chart without problems.


So, there's another example of how to waste a (actually another) few hours by using eSignal. Are these people being paid by another group of traders so that we can't trade properly, I don't get it.

Anyone know a way to port my efs to any other charting app?

eSig v10.0.1086.932


Proto Local Address Foreign Address State
TCP first07:1541 ip234-119.esignal.com:2189 ESTABLISHED
TCP first07:1544 ip232-216.esignal.com:http ESTABLISHED
TCP first07:1548 ip225-131.esignal.com:2194 ESTABLISHED
TCP first07:1653 ip232-46.esignal.com:2192 ESTABLISHED

[eSignal_JasonK]

Hello Dave,

Thank you for your detailed post and feedback. We do understand and acknowledge that our tick charts performance in version 10.0.1086 has some inefficient behavior. Improving this aspect of the application was one of the primary areas of focus in our 10.1 version, which is currently available as a release candidate. Please feel free to try the latest version, which is available on the eSignal download page. I think that you will find many of the scenarios you’ve described have already been greatly improved.

The no TS message indicates you were having a connection issue with the farm. If there was no history being displayed, you may have lost connection to a history server. When this occurs, restarting eSignal will reset all your connections. If you continue to have connection problems, please contact our TS department and have them trouble shoot your connections.

One of the problems we’ve addressed in 10.1 is how tick requests were being made. In 10.0 and previous versions, tick data was requested in whole day segments. As the amount of data from the exchanges has been increasing over time, this amount of data takes longer to process at the desktop. In 10.1 we are no longer making whole day tick requests. The requests are broken up into smaller requests of 200 ticks at a time so that the desktop can process the data and better manage memory/cpu usage. Also, if the time template is in dynamic mode, the program will download the amount of bars needed to fill the chart depending on the bar spacing and size of the window with an approximate minimum of 200 bars. Previous versions were forced to load a much greater number of bars, which was more than what was needed in most cases. It will still take some time to download large amounts of tick data for time template settings that define such requests, but the application performance is much improved over previous versions. Tick charts also now support start and end times in the time template settings as well as the number of bars option. These settings were not supported in 10.0 and previous versions.

One change in tick chart behavior to be aware of is how cached data is being utilized. We had to make some changes here in order to utilize the newer components for the data requests. The tick caching is still present, however, it is specific to the intervals requested. In other words, if you request 10000V of ES and then request 20000V of ES, there will be a separate cache for both intervals. The 20000V will not use the 10000V tick cache (or data downloaded via Tick Downloader). If you go back to 10000V from a different interval, the chart will load the cached data for the 10000V set and not re-request the data from the farm.

[Dave180]

Jason,
Many thanks for taking the time to explain these changes. Although I'm not sure they directly apply to the odd repaint problems I discovered this time (ie even when the data feed was deliberately disconnected) it is nice to know the things that you are changing.

A couple of statements you made seem a little "odd", perhaps you could elaborate:

if you request 10000V of ES and then request 20000V of ES, there will be a separate cache for both intervals....and not re-request the data from the farm

Surely the Volume Charts are constructed from tick data (after all an EFS running on a tick-bar chart receives each trade tick). So I had assumed that the cache is, and always would be, a "tick" cache and is unrelated to the type of tick-bars (be they any size of Volume or Range and Price even).
When I currently (10.0) change period from 20000V to 10000V there is no obvious new tick download. You seem to be saying that in 10.1 there will be a new download for each initial change of periodicity on a tick-bar chart. I don't understand why there would need to be separate tick downloads for different periods, in fact it seems extremely (unbelievably) inefficient to me to download the tick data again - and then there is the local memory cache (I assume) needed to store separate caches if you regularly flip between different volume sizes.

This also suggests that displaying a, say, 2R then a 3R chart would also request the whole tick data set twice more? Changing the size used with V and R is quite common Please explain why you would want to wait to re-download the tick data for an initial view of all these different periods.

What would be really useful would be a local tick cache that persists between eSignal instances, I can't understand why this isn't seen as a natural fix to reducing bandwidth?

For charts that use long-term (multi-day) templates (because of largish Volume bars and long-period indicators), and thus always need a lot of tick data, will the actual download and chart start-up be any faster?

I'm afraid that until the number of bug posts about 10.1 dies down it is not viable for me to try using it, especially as I see there are some outstanding EFS issues.

Regards

Dave

[eSignal_JasonK]

-

Hello Dave,

You're most welcome. Please allow me to further clarify a few things.

Regarding the repainting issue, there were a couple known painting problems in various windows, including the advanced chart, which have been fixed in the 10.1 version.

When I refer to tick cache, I'm referring to cached data for tick charts. Tick data is still cached, but the chart can only utilize the cache for intervals that were previously requested. As I explained in my prior reply, we had to make some changes here in order to utilize newer components for making the tick requests to improve that process. As we move forward we plan to make improvements in this and other areas to address bandwidth considerations in future versions.

[Dave180]

Hi, thanks Jason

but the chart can only utilize the cache for intervals that were previously requested

Just in case "intervals" may have some meaning I don't understand, I'd just like to be absolutely clear here.
If, in 10.1, I draw an ES 10day 1R chart, then a 10day 1.25R chart, then a 10day 20000V chart will I end up downloading the whole 10day ES tick data three times, ie twice more than in 10.0?
If the answer is Yes" then I really don't see this as a workable product. Apart from anything else my ISP (a high quality one that charges me for exceeding a limit) will be rubbing their hands with glee.

I mean, it is yourselves that remind how much more tick data there is nowadays and the rate at which it growing and how that causes problems.

Can you give some stats re the downloads:
either bytes per (1K?) Trade and Quote tick(s), or
a multiplier to apply to the size of the .epf text file that results when the tick data is saved (I assume the download is compressed in the download)
relative download time vs 10.0 (and is that dependent on the bandwidth)
The time that 10days of ES tick data should take to download in with 10.1 (it takes several minutes in 10.0 based on a connection that runs at about 4Mbps (an ADSL"rated" at 8mbps here in UK)

Thanks

[ACM]

Dave180

If, in 10.1, I draw an ES 10day 1R chart, then a 10day 1.25R chart, then a 10day 20000V chart will I end up downloading the whole 10day ES tick data three times, ie twice more than in 10.0?

That is correct

If the answer is Yes" then I really don't see this as a workable product.

As a user I find instead that the changes implemented in 10.1 for tick based intervals make this a much more workable product than any of the preceeding versions. For me up to now the biggest limitations with any of the tick-based intervals have been that I had to always download these intervals in whole day segments [even though I rarely use more than 300 bars for any of my trading] and that I could not set the Start/End times. With version 10.1 I can now do both which drastically lowers the download times for me (and I believe for a multitude of other users).

I mean, it is yourselves that remind how much more tick data there is nowadays and the rate at which it growing and how that causes problems.

I can't speak for eSignal but I believe that the changes they have implemented help in resolving exactly that problem by allowing the user to download a specific subset of data. Through the years most of the posts I have seen in this forum on the issues of tick based intervals have been about setting Start/End times and being able to download a user defined number of bars.
Version 10.1 responds IMHO to those necessities. As Jason said however this means that tick based intervals are now cached by specific interval a small price to pay in my opinion given the huge advantages that the changes they have implemented offer to many.
Regardless as Jason indicated this is a first step. It is my understanding that there other changes in the works that should ultimately make tick based intervals even easier to use and even less resource intensive
Alex

Originally posted by Dave180
Hi, thanks Jason

Just in case "intervals" may have some meaning I don't understand, I'd just like to be absolutely clear here.
If, in 10.1, I draw an ES 10day 1R chart, then a 10day 1.25R chart, then a 10day 20000V chart will I end up downloading the whole 10day ES tick data three times, ie twice more than in 10.0?
If the answer is Yes" then I really don't see this as a workable product. Apart from anything else my ISP (a high quality one that charges me for exceeding a limit) will be rubbing their hands with glee.

I mean, it is yourselves that remind how much more tick data there is nowadays and the rate at which it growing and how that causes problems.

Can you give some stats re the downloads:
either bytes per (1K?) Trade and Quote tick(s), or
a multiplier to apply to the size of the .epf text file that results when the tick data is saved (I assume the download is compressed in the download)
relative download time vs 10.0 (and is that dependent on the bandwidth)
The time that 10days of ES tick data should take to download in with 10.1 (it takes several minutes in 10.0 based on a connection that runs at about 4Mbps (an ADSL"rated" at 8mbps here in UK)

Thanks

[Dave180]

I'm updating this thread with my experience of using 10.1 with multiple day ES #F Advanced Charts with Volume (like 1000V up to 50000V).

Basically, it's a disaster. Start with a 40000V chart of ES #F, scale the bars so you see two or three days of data.
Now change to a 50000V chart, see the chart painfully redraws as the data is downloaded (and my ISP rubs their hands in glee). Change again, repeat.

PLEASE eSignal just implement some local tick cache (preferably with a life of longer than an eSignal session). Surely you can find a way of having your new "per-periodicity" cache AND a "raw tick" cache at the same time.

To me it just seems unbelievable that despite having a goal of reducing tick downloads which you you address by using sub-day sized download chunks you completely blow it by now forcing multiple-day downloads to be repeated just because the bar size changes.

[eSignal_AveryH]

Hello Dave180,

The eSignal application does cache data using the user defined time templates feature. Much like the discussion you had with ShaheedM in this use.post to use user defined time templates for seconds or ticks, you can edit your time template to use volume and to request the maximum amount of data that you might use.

This results in a lengthy initial download, but once complete other volume type charts you request (40000V, 50000V) are very fast because they are created from the cached data.

AveryH
eSignal Support