serious problem of missing data

[Rambo]

There is a serious problem of missing data both for historical and real time advanced charts for Japan market, sub-minute time intervals.

For example, for $topix-tse, quarter of a minute time interval (15seconds), 10 data points are missing for the first 20 minutes this morning from 9:00 to 9:20, exchange time. The following are the missing points.
9:09:00, 9:11:00, 9:13:30, 9:15:15, 9:15:30, 9:15:45, 9:16:00, 9:20:15, 9:20:30, 9:20:45.

The problem is just as bad throughout the day and every day.

Could the moderator please pass on the information about this problem to someone who can address it? Thank you.

[jay60]

Re: serious problem of missing data

Originally posted by Rambo
There is a serious problem of missing data both for historical and real time advanced charts for Japan market, sub-minute time intervals.

For example, for $topix-tse, quarter of a minute time interval (15seconds), 10 data points are missing for the first 20 minutes this morning from 9:00 to 9:20, exchange time. The following are the missing points.
9:09:00, 9:11:00, 9:13:30, 9:15:15, 9:15:30, 9:15:45, 9:16:00, 9:20:15, 9:20:30, 9:20:45.

The problem is just as bad throughout the day and every day.

Could the moderator please pass on the information about this problem to someone who can address it? Thank you.

>Rambo

Using a 15S Advanced chart for $topix-tse the data is displayed properly on my charts.

[Rambo]

you need look at the time stamps

Hi, Jay:
You need to look at the tme stamps or put 1 minute vertical grid on in order to see the gap.

Attached are two advanced charts of 15s interval with missing data, with 1 minute vertical lines. Without missing data, there should be four points between the lines.

[Rambo]

2nd chart

2nd chart

[jay60]

>Rambo

Actually, for whatever reason, my chart is slightly different from your chart. As you say, but at different times on mine, there are missing ticks. As the Topix is priced every 15 seconds, maybe two ticks are included in the bars at 14:42:00 and 14:43:45.
Attached is a chart from a Japanese broker for the same time period as one of your charts.

[Rambo]

hard to believe extensive gaps in original data

Jay:
Thank you for posting the chart from a different source. I can see the missing points on the chart between 14:41 and 14:43, same as in my chart from Esignal. But I am not sure this is enough to suggest that the problem comes from the exchange. It could be just a coincidence, for three reasons.

1. For 14:43 to 14:44, your chart does have all the data, while my chart from Esignal shows missing data.
2. There are just too many missing data, one every two minutes on average, to think that the problem is entirely caused by the exchange.
3. I did not remember having such an extensive problem before when using a different data source.

I would hope someone from Esignal could check with the exchange if the original data come with extensive gaps.

[jay60]

Re: hard to believe extensive gaps in original data

Originally posted by Rambo
Jay:
Thank you for posting the chart from a different source. I can see the missing points on the chart between 14:41 and 14:43, same as in my chart from Esignal. But I am not sure this is enough to suggest that the problem comes from the exchange. It could be just a coincidence, for three reasons.

1. For 14:43 to 14:44, your chart does have all the data, while my chart from Esignal shows missing data.
2. There are just too many missing data, one every two minutes on average, to think that the problem is entirely caused by the exchange.
3. I did not remember having such an extensive problem before when using a different data source.

I would hope someone from Esignal could check with the exchange if the original data come with extensive gaps.

I do not really have much knowledge in these matters and should not be commenting. However, if you look at the T&S, you will see the quote is not always printed exactly every 15 seconds. Therefore. I would guess that all the data is there on the eSignal chart but two or more 15 second ticks are included in the those 3 bars on your chart as it would take more than one tick to form a bar. But at the same time, for example, there is no quote for 14:37:45 on the chart on in T&S.

I don't use 15s charts. But for the 1 minute etc. charts that I use, I've found eSignal's data to be extremely accurate when compared to that of local brokers, Reuters etc.

[eSignal_AveryH]

Rambo,

To better resolve data issue like this we recommend that you contact Technical Support via LiveRep at the time you're experiencing the issue. The Technician can then check the servers you're connected to and test then for data accuracy while you're still connected and report any problems seen.

LiveRep can be accessed from Support-> LiveRep on the main menu of the eSignal application.

Trying to troubleshoot after the fact is much more difficult as in many case your connection logs have been overwritten and we do not know where you've been connected to check.

I did however pull up the $TOPIX chart and the data matches that of the Time and Sales window. Please see that screenshots of the chart and T&S Window.



If you'd like I'll try and locate your earlier connections. Please PM the username I'll be searching on.

AveryH
eSignal Support

[Rambo]

Thanks

Avery:
Thanks for looking into the issue. My username has been sent to you by PM.

Rambo

[Rambo]

Data gaps in advanced chart

Hi, Avery:

As you suggested, I checked the data sheet of advanced chart for $topix-tse, 15s, against time and sales data. While the time and sales data are complete, one data point every 15 seconds, there are 6 gaps in the data sheet from advanced chart for the 10 minute period. The advanced chart failed to pickup the data already stored in Esignal system. Attached are the two data sheets.

Thank you.

Regards,
Rambo

[Rambo]

Time and Sales

The data from the time and sales sheet attached below are complete, with no gaps that appeared in the advanced chart.

[ScottJ]

The calculation for say a 15 second chart (15s) is not 100% precise. Between slight delays from the source to even minor latencies on the desktop, differences in milliseconds can cause an update to slide to the next bar.



As you can see in the highlighted box above, there are two updates w/in the 14:48:30 bar, including the 851.13 price that theoretically could have made up the next bar at 14:48:45.

If the exchange is only sending out 1 update every 15 seconds, you can simply use 1T for tick as your interval and get every update.

Thanks.

[Rambo]

1T shows gaps too

Hi, Scott:

I tried to use 1T interval. It displayed similar data gaps as 15s interval.

Desktop latency is unlikely to be a cause as live data show the same pattern of gaps as historical data, which are stored by Esignal.

As you mentioned, it is a problem of the advanced chart (15s or 1T) not being able to recognize data bearing time stamps with one or two second difference from 15 second multiples. Hopefully, the issue can be raised with the programmers. I would think this is not a difficult issue to address.


Many thanks for your help.

Kind regards,
Rambo