Why is the Tick Downloader skipping data?

[SteveH]

I'm running version 10.5.1721.1140 with Windows 7 64bit setup for XP SP3 compatibility mode (per the knowledge base instructions) and the Tick Downloader is skipping data on all my symbols.

Here's a sample from a download of the "TF #F" tick data

There's huge data holes like this:

Q,091203,235916,0,588,0,2,,
Q,091204,152215,0,600.8,0,8,,

All of my data skipping is doing this midnight to 15:00 to 15:30 time area skip for every day for the last 10 days.

I'm skipping from 125917 to 152214...say WHAT?!

My clock is correctly set to Eastern Standard Time and I do not sync the clock with any time server. I have the optional time span fields empty and trades only set to unchecked.

This is happening on all of my e-mini tick data downloads but not on NYSE-based symbols. Things like downloading the $tick data are just fine.

[SteveH]

I just ran a VMware session with Windows XP SP2 on EST and an eSignal 10.4.x version and I downloaded the last 10 days of the TF tick data.

Here are all of the holes:

Q,091201,235945,0,588.7,0,1,,
T,091202,144009,594.6,1,

Q,091202,235955,599.2,0,8,0,,
Q,091203,153003,593.2,0,17,0,,

Q,091203,235916,0,588,0,2,,
Q,091204,152215,0,600.8,0,8,,

Well, now I'm going back over all of my archived e-mini tick data download files and here's the pattern since the beginning of November 2009 (my files from October 2009 and older appear to have no holes)

The oldest 3 days of a 10-day tick download have the same kind of data holes as you see above while all of the other midnight crossover dates in the file don't have these missing data gaps.

eSignal support, could you please look into this for the TF #F and ES #F symbols on the tick download?

Thanks.

[SteveH]

One last thing. I download the e-mini tick data files every week so there's overlap of my data downloads since I always pull the last 10 days.

In the last 2 months, if midnight crossovers overlapped between download files AND the midnight crossovers weren't among the oldest 3 days in the file, then there wasn't a data hole.

For example, I have 2 data files for the TF #F symbol named:

TF Nov 25th and 10 days back.EPF
TF Nov 18th and 10 days back.EPF

The midnight crossover date of 12/12/09 to 12/13/09 in the "TF Nov 25th..." file has this hole:

Q,091112,235851,0,581.5,0,6,,
T,091112,235941,581.3,2,
Q,091112,235941,581.3,0,3,0,,
Q,091112,235941,581.3,0,6,0,,
Q,091113,145912,579.8,0,12,0,,
Q,091113,145912,579.8,0,2,0,,
Q,091113,145912,579.8,0,12,0,,
Q,091113,145912,579.8,0,2,0,,

BUT, that same crossover is found in the "TF Nov 18th..." file and there is no data hole because it wasn't a midnight crossover date which was one of the 3 oldest days in the file:

Q,091112,235851,0,581.5,0,6,,
T,091112,235941,581.3,2,
Q,091112,235941,581.3,0,3,0,,
Q,091112,235941,581.3,0,6,0,,
Q,091113,000315,581.4,0,1,0,,
Q,091113,000315,0,581.5,0,5,,
Q,091113,000315,581.4,0,2,0,,
Q,091113,000315,581.4,0,3,0,,

This should be reproducible by eSignal technical support. It's been going on since the beginning of November 2009.

[shortandlong]

I'm having the same problem - gaps of up to 12 hours of ticks missing. I usually download 10 days of tick data over the weekend.

For example, 2 gaps for GC G0=1 downloaded 12/13/09:

Q,091202,235954,1218.5,1218.6,1,2,,
...missing 12+ hours of ticks...
T,091203,122930,1218.4,3,

Q,091203,235957,1205,1205.2,5,5,,
...missing 12+ hours of ticks...
Q,091204,124314,1170.3,1170.6,14,2,,

I'm using the latest version of eSignal, 10.5.1721.1140 (9/17/2009), on a Vista 64-bit Core i7 machine. Time zone is CST.

[shortandlong]

I think the key to reproducing this is requesting the data over the weekend, or maybe a Friday or Monday.

[shortandlong]

Went through my data to "assess the damage". The problem first occured for me on October 24th (a Saturday). The weekend prior to that everything was ok.

Note at the time I was running eSignal 8.0 on a Pentium 4 with XP SP3, and EST time zone. So, it occurs on both my new and old system.

The problem seems to be with data after midnight with 2 intervening weekends between now and then.

Just requested CL F0=1 on both systems:


eSignal 8.0:

; Symbol=CL F0=1
; Date=12/04/09-12/15/09
...missing 14+ hours of data for the first day...
Q,091204,141109,75.05,75.06,2,9,,
Q,091204,141109,75.05,75.06,3,9,,


eSignal 10.5:

; Symbol=CL F0=1
; Date=12/01/09-12/15/09
...
Q,091201,235951,78.52,78.55,4,3,,
...missing 12+ hours of data...
Q,091202,124659,76.84,76.86,1,15,,

Q,091202,235958,76.76,76.77,2,3,,
...missing 12+ hours of data...
T,091203,122528,76.89,1,

Q,091203,235959,75.96,75.97,1,2,,
...missing 12+ hours of data...
T,091204,125320,74.91,4,

[shortandlong]

Tested downloading HO F0=1 today.

The tick downloader is missing data even when a full 10 day tick chart of the same (correct) data is displayed. (Actual chart was HO F0=1, 10T - dragged until the end of data on 12/2/09.) So, I don't think this is a server problem, but a bug in the eSignal client tick downloader code.

Also, it isn't necessary to have 2 intervening weekends to reproduce the problem.

HO F0=1 missing data (eSignal 10.5, CST):

; Symbol=HO F0=1
; Date=12/02/09-12/16/09
T,091202,170001,2.04,1,
...
Q,091202,235736,2.0403,2.0417,2,1,,
...missing 10 hours...
Q,091203,100606,2.05,2.0503,2,1,,
...
Q,091203,235959,2.0384,2.0398,1,1,,
... missing 10+ hours...
T,091204,104641,2.0353,3,
...
Q,091206,235945,2.0365,2.0372,3,1,,
...missing 11+ hours...
Q,091207,110314,2.0099,2.0108,1,8,,
...

[SteveH]

eSignal Support,

You've got 2 independent observations now. If we're losing data in our tick downloads than many others probably are as well.

It's time to chime in here on this thread and let us know that you're looking into this problem.

Thanks.

[ScottJ]

We will take a look and let you all know. We appreciate the specific findings.

Thanks.

[eSignal_AveryH]

We're continuing to investigate but we're certain that it is not ESignal or OS version specific. In our troubleshooting we're able to duplicate what you've reported on Window XP as well Windows 7.

We've added some of our Network Team to now check on the server side on how the request is being made and processed. Once this portion of testing is complete we'll follow up.

We thank you for patience while we continue to investigate the problem.

[SteveH]

Thanks Avery.

From our combined reports on the data we've been saving, it appears that some software change, whether it be server or client side, was made in the last week of October 2009 to cause this anomaly.

[shortandlong]

Thanks Scott and Avery for letting us know that eSignal has been able to reproduce this.

The fact that data is NOT missing from tick charts, but only the tick downloader, should help isolate the problem.

Also, I would suspect code related to daylight savings time as a possible culprit, given the Oct./Nov. timeframe when this started occurring and that data after midnight is missing.

The other possibility is a descripancy in the code between 10 business days and 10 calendar days when requesting 10 days of tick data. Most likely, most of the code is using 10 business days, but somewhere 10 calendar days is being erroneously used.

[eSignal_AveryH]

Thanks to everyone for the patience and input, we've discovers some issues with the way the downloader is making large request to to the servers. This information has been passed back to the developers to continue working on.

We'll continue to follow up and once we get more information we'll post back.

[SteveH]

Has this been fixed in 10.6 and I need to upgrade to get it working?

I would think that it's considered a pretty big deal that your tick data downloader is creating huge time gaps and making 30% of any 10-day download worthless for playback.

[eSignal_AveryH]

There is a meeting with the Product Manager and Developers to see what progress has been made and what still need to be done to correct the tick downloader. Once we have more information we’ll post back.

We thank you for continued patience.