Data Integrity - Splits, gaps, spikes, whatnot.

[Larry Marchman]

Originally posted by JayF
Got lucky, one senior network operator walked by, and indicated that while there was a large surge in bandwidth at that time, we didn't reach any limits or caps that prevented data from going out. He also said that the likely cause of the pings jumping up was that UDP / ICMP packets used by the ping command are the first packet type to be dropped or delayed during high usage.

I didn't experience what Carol reported (Not Responding state). Just the usual FedSpeak data delays. Without being too snarky here, I would suggest the Network folks review the accuracy of the reporting mechanisms they have and use to subsequently state whether or not servers are stressed and delaying delivery of data. I understand the dropping of Ping packets in an ordered priority hierarchy. But delayed data was afoot yesterday, without doubt.

Originally posted by JayF
I looked at the eSig forum and didn't see any reports of issues from yesterday. Larry, could you clarify what you were seeing there?

Here's one thread as an example. Be sure and scroll back to the posts for yesterday 4/30 and the ones after FedSpeak...

http://forum.esignalcentral.com/showthread.php?s=&threadid=26541

or for a specific post to start with...

http://forum.esignalcentral.com/showthread.php?s=&postid=114401#post114401

LAM

[lugnutz]

Carol,

I've witnessed a number of dropped data packets on eSignal's side during normal operation, with a *huge* spike in packet errors and drops when loading up a new symbol. You may have hit the perfect storm of a "normal" problem that became a critical problem when the market was particularly busy. (BTW, I've seen packet loss with 5.x as well as 6.x)

I'd spend more time characterizing the packet loss, but eSignal hasn't responded or acknowledged either of the posts I've made on the subject already, so why bother?

lugz

[tradertom]

Hey Lugz,

I am not an eSignal employee but I saw the previous posts and was curious (though I am not network-savvy at all!) Is the packet loss something that can be fixed? I was thinking that packet loss was due to the servers on the route between the sender and receiver somehow dropping them and requesting a resend causing delays and frustration. Are you saying that there is a tweak that eSignal can make on their servers to prevent packet loss?

Thanks,
Tom

[lugnutz]

Hey Tom,

The reason it looks like a problem on eSignal's side is that the actual amount of packets created by a single user is relatively small on the internet scale. Even with a large workspace, my network and CPU utilization is very, very low.

Now you could speculate the packets are being lost somewhere in the internet cloud, and certainly a small percentage of them are, but when I saw over 100 dropped packets in a minute... that's suspicious. Combine that with the data other folks have posted here - including ping traces that show the path all the way to eSignal is fine... and you've got a pretty compelling story.

Can it be fixed? That depends on the exact problem. It could be that whoever eSignal gets their network pipe from is capping eSignal's peak network utilization to preserve quality of service for other customers. If that's the case, eSignal could pay for additional bandwidth and/or open additional data centers to try to better distribute the load (instead of consolidating data centers which focuses the load.)

Another possibility is that the bandwidth capping is done on a per user or per connection basis... that may better explain why so many packets are lost when loading new symbols causes a brief burst of packets to swarm.

A simple test would be to do a packet capture at 3AM when the world is quiet.... but I just haven't cared enough to try it.

lugz

[tradertom]

Thanks for the explanation lugz. I am generally pretty handy with computers but the networking stuff is always confusing to me (what can I fix on my end versus what can they fix on their end versus what cannot be fixed due to the architecture of the internet.)

Best regards,
Tom

[tradertom]

$INDU is painting a rogue candle dated January 16, 2009 on Daily and Weekly and Monthly charts.

On QCharts 6.0.2.

Can you guys please remove this false candle?

Thanks,
Tom

[bpschultz]

Forex Data Problems

The following was originally posted elsewhere on 4.28.08 at 10:02 a.m. I have not received a response from QCharts personnel, nor have the problems been addressed.

I am looking at 60-minute, all sessions charts of XAU A0-FX and XAG A0-FX. It is obvious that QCharts 6.0.2 does not use the correct open time on Sundays. I can see this same data from GTIS on other data providers, and those markets open at 18:00.

In addition, the daily snapshot bar on my XAG A0-FX chart shows the high is 17.04 and the low is 16.81. However, the 60-minute bars show the low is 16.77. According to my other data provider of GTIS prices, the daily low is 16.77. This value is also consistent with the COMEX futures price.

Finally, the high for the 4:00 bar of XAG A0-FX is 17.06 rather than 17.04. Apparently, QCharts missed several ticks during that timeframe.

Please adjust the mkthours file and/or whatever else is necessary to ensure that QCharts collects and correctly charts all data from the open to the close each day of the week.

I would appreciate a prompt resolution to these problems. Thank you.


__________________
Brian Schultz

[Larry Marchman]

Forex Data Problems

[Edit addition...or you could use the COMEX:GC RTH/ETH mapping if you wish since looks like the FX quits at 17:15 thru 18:00 each day to mirror COMEX]

Hey Brian,

Regarding your FX Metals issue...
The FX mapping isn't in MktHours.inf. Rather it's in MktHoursESignal.inf. I'm not sure who does the MktHoursESignal file or where it's downloaded from.

In the interim, until the issue is addressed by whomever, and what with you being a MktHours maven, and if you haven't done a local tweak already <grin>, you could put this in a MktHoursESignal.UNF. This will map from Sunday 18:00 thru Friday 17:15. Looks like the FX Metals quit at 17:15 on Friday. And looks like FX has no holidays.

;{eSig};=18010101{32[-18:00,20:00,1]}{28[-20:00,20:00,1]}{2[-20:00,17:15,1]}
;{eSig};:FX:XAU*,FX:XAG*

Also fwiw, looks like the FX Currencies open even earlier than 18:00 Sunday, but it's not a hard solid open like the Metals. It's all over the place differing Sunday to Sunday. Since there are no "formal" definitions of FX hours, not sure about what the coding would or should be for the FX Currencies.

The above tweak assumes you're not using other Exchanges in the MktHoursESignal file. The UNF does the usual over-ride of the INF and any subsequent changes for the INF will not be picked up. Ya know, like a holiday for some other country.

LAM

[bpschultz]

Forex Data Problems

Larry,

Thanks for the tweak! I'd looked at that file but my attempt to modify it didn't work. I'll give your code a try this evening.

Brian

[bpschultz]

Forex Data Problems

Larry,

The tweak worked! Muchas gracias for your time and effort.

Now we just need someone at QCharts to devote a little time and effort toward developing a correct version of the mkthoursesignal.ini file so that users of v6.0 can receive complete and accurate data.

Brian

[Larry Marchman]

Re: Forex Data Problems

Originally posted by bpschultz
Larry,
The tweak worked! Muchas gracias for your time and effort.
Now we just need someone at QCharts to devote a little time and effort toward developing a correct version of the mkthoursesignal.ini file so that users of v6.0 can receive complete and accurate data.
Brian

Cool. Thanks for the Thanks, Brian.

Jay...
Assuming here someone will get the FX Metals updated in the MktHoursESignal.inf.
And the FX currencies should be looked at also, methinks. They start before 18:00 on Sunday. Looks like, as a kinda pattern, about 15:00 (sometimes 14:00) as the time when volume starts to pick up. Current coding has them start at 20:00 Sunday.

LAM

[jmglkg]

SPX, VIX, VXN

On every server I try, there is a gap in data for SPX, VIX and VXN for Monday May 5th.

I know CBOE had an outage, but is it possible to get that data populated?

Thanks.

[grunewaj]

Error in the APC Daily

Yesterday's low for APC was $73.01 according to Yahoo and the shorter charts, but the Daily for APC shows a low of $68.14. Could somebody please fix this?

Thanks,

Jim

[JimRutherford]

NOV Daily candle QC 5.1

5/8 candle on the daily chart QC 5.1 has an incorrect closing price. Sources are QC 5.1 233 chart, QC 5.1 quote sheet, Yahoo and brokers. The closing price as shown on the candle is off by about $2

Jim

[ckelly]

CLF split this morning. The Intraday's look good - the Weekly/Daily charts need some help.

Carol