more bad ticks than before

[clearpicks]

Hello AveryH,

I am still experiencing the frequent bad tick problem (one to four bad ticks for each futures contract I watch/trade, especially for NYMEX crude oil and natural gas contracts). Could you tell me which ones are true in the following cases:

1. This is a specific issue related to the commmunication between eSignal v10 data manager and eSignal data servers.

2. This is an issue related to any version of eSignal data manager and eSignal data servers.

3. This is an issue of existing eSignal data server and this will be automatically fixed after migrating to the new data servers.

4. This is an issue of eSignal data servers and would not be automatically fixed even if the migration to the new data server bank is completed.

4. This is an issue of NYMEX exchange datafeed and eSignal could not easily fix it.

5. eSignal engineering department does not know the reason but has put it on bug tracking list.

6. eSignal engineering department has reproduced the bad tick problem but not put it on bug tracking list yet.

7. eSignal engineering department has not done anything related to this issue yet.


Your response would be greatly appreciated.


- Clearpicks

Originally posted by ahodgee
Clearpicks,

We understand your frustration with bad ticks, and we are making every effort to reduce or eliminate them. Until we can accomplish this we have mechanisms in place to assist with correcting bad ticks in a timely manner. Tools like tick filtering in the charts, Bar Editor and the Bad tick Reporter. We encourage you to use these tools for correcting and reporting bad ticks until we have implemented the necessary improvement on our data feed to better filter them out. The forum is probably not the best place for report bad ticks you’re seeing.

Thank in advanced for you patience and understanding.

AveryH
eSignal Support

[eSignal_AveryH]

Hi Clearpicks,
Thank you for the post and for your patience while I try to best explain what eSignal and the Exchanges are doing to correct bad ticks. I was hoping to get a screenshot for back ticks on CL but today seemed to be a good day at the NYMEX. (Murphy’s Law I guess).

In the coming months we will be rolling out the new CID tick server which will show Sequence number for every tick. This also makes it possible for the users to see when the Exchange or eSignal make corrections for bad ticks. The corrections will be instantaneous for advanced chart user,m however, users who continue to use the Standard Charts will see no difference.

This first screenshot is where the exchange deleted a tick on 11/2/2007, under the conditions column it confirms it as a delete and it is also color coded for any corrections, inserts or deletes.


This screenshot shows an insert of a price and a delete on the bid, so it will not be limited to just bad ticks on trades but also on bid/ask. Notice the change between this Time and Sales on the Time and Sales connected to a regular Tick Server.



We hope to have this rolled out in the next couple of months, a lot of testing is still yet to be done. We ask that you continue to be patient as we get this new infrustructure in place.

AveryH
eSignal Support

[clearpicks]

AveryH,

Thank you for explaining the CID thing. It seems a good solution to get a quick fix of back tick without going through the tick report process or manual back tick correction.

One thing puzzles me is that why the bad ticks of CL I saw recently all appeared in charts with interval higher or equal to 1min but not in second charts. If those ticks were sent out by the exchange, they should appear in all time frames. Because I donot watch time&sales window, I am not sure whether those bad ticks appear in T&S window or not. However I remember at least in a few cases I checked they were not there. (I think all charts with time interval lower than 1 minute are built from raw tick data by the data manager which are the same as those in T&S window.) If this is true, can the CID thing correct bad ticks in minute charts when no bad ticks in T&S window or second charts?

If the CID thing does be able to the correct bad tick in 1min charts, do I need to modify the EFS to make it aware of any correction or the EFS would be automatically reloaded?

Best Regards,
Clearpicks

[clearpicks]

If you view today's CL F8, at 10:31, the high of 1min bar is not correct (94.80) and there is not matching trade in both 10sec chart and T&S window.

Another bad tick at 10:50 (bar's low with wrong price 92.10).

[eSignal_AveryH]

Clearpicks,

This is a good example of where the new servers will help, The attached the Time and Sales shows the 94.80 trade was deleted at 7:31:18 PST (10:31:18 EST).



To your question regarding seeing ticks on the Minute charts you don't see on the seconds; it going to depend on a couple of different factors,
1-If looking at the seconds chart, and the time you want to see is not actually visible on the chart, and it require you to scroll back you are essentially refreshing the data on the chart which is removing that back tick.
2-If you see a bad tick on the minute chart and switch to a seconds chart, you have refresh the chart and removed the back tick.

Regarding the EFS I'd have to get with our QA and EFS teams about how EFS will handle bad ticks. I'll get more information and get back with you.




AveryH
eSignal Support.

[clearpicks]

AveryH,

Thank you for answering my questions.

Can the new data server handle the deletion of bad tick automatically? The thing I don't understand is that currently the bad tick only affects the charts with interval equal or higher than 1minute and forced reloading data or restarting eSignal can not remove the bad tick unless someone already did a bad tick report. Since each new incoming tick (including bad ticks) is used to incrementally build candlestick bars, why did it happen that way? One possible scenario might be
the eSignal server I connected to is already capable to delete bad ticks using the CID thing, however the thread building intraday bars can not fix the incorrect high/low of old bars even if that bad tick has been deleted (thus, restarting eSignal did not help but the sub minute charts and T&S were corrected). Is this true? Hopefully on the new data server the thread building historical OHLC data serials for intraday symbols can handle the deletion of bad ticks so that I only need to ctrl-click the "OK" label to refresh the charts without the unpleasant distraction of going through bad tick report process.

- Clearpicks

[clearpicks]

I believe those bad ticks are caused by the server's lack of capacity to handle volume spikes because almost all spikes I observed occured at times or within short time after trading activities picked up.

[shortandlong]

It's worse than that...

Having said that, I don't believe this is eSignal's fault.

It appears the prices NYMEX/COMEX are supplying are from the combined pit and Globex sessions.

I bet during fast markets (where most of the bad prices occur) pit traders are in too much of a rush to punch in the correct price manually.

The reason it's worse than that is the prices you're watching don't represent the market you're trading!

If you don't believe me watch the last trade vs. the bid/ask. Very frequently the last trade is outside the range of the bid/ask! This is especially true of gold (GC G8) and crude (CL F8). If you're placing Stop or Limit orders it's almost impossible to tell if you've been filled.

I've requested in another thread ("Suggestion for all the bad ticks on NYMEX/COMEX " in eSignal forum, here) that eSignal separate the pit and Globex prices into different symbols to resolve these issues.

[clearpicks]

Hello shortandlong,

I was talking about those bad price spikes on intraday charts which is abvious out of range (for example, one or two points away from the previous tick and immediately retrace back to the correct quote, which leaves a very long fake wick on that candlestick. The thing puzzles me most is that if I open a 1min chart and a 10sec chart of the same instrument, when the fake spike appears on 1min chart, in most cases it does not shown on 10sec chart. If eSignal data manager uses the same realtime tick-by-tick data stream to feed the two charts, why 1min chart shows the bad tick while 10sec chart does not? If we assume eSignal data manager open two connects with the data server, one for feeding 1min chart and one for feeding 10sec chart, then either the bad tick caused by data corruption during transmission or it is caused by error on the server side. Because it is very likely the NYMEX exchange only feeds eSignal's data server with tick-by-tick raw data, and it is eSignal's data server's responsibility to build 1min data sequence, if the bad tick is not cuased by data corruption during transmission, it must be error on eSiganl server side causing that bad tick. Given the extreme high reliability of data transmission nowadays, the later case is most likely to be true. Unfortunately I have not received any serious explaination or investigation on this issue.

The bad tick you talked about was trades out side of bid and ask range. But in most cases it is only just a few ticks away so I don't case much about it. The situation like yesterday evening is very rare.

- Clearpicks

[shortandlong]

Clearpicks,

I've found the true number of bad ticks doesn't really show up on bar charts (candle or otherwise) for NYMEX. The tick charts I use shows bad ticks of all sizes. (For CL F8 I use 1200T, 300T, 50T, and 15T charts.)

The charting software I use ignores eSignal's tick corrections and caches all ticks, so I see the data as it was sent by eSignal and can review/edit the Time and Sales on my PC.

On an average day for CL F8 I get:

1-5 bad ticks in the 1 point range

5-20 bad ticks in the 10-30 tick range

I suspect the number of bad ticks below the 10 tick range is in the hundreds...

I subscribe to NYMEX, COMEX, CME, and CBOT. I almost NEVER have bad ticks on the CME or CBOT.

I would like to see eSignal have the following priorities in resolving this issue:

1. Go after the source of the problem rather than fixing it up after the fact. Cause: Pit traders hurriedly punching in partial prices during fast markets. Solution: Eliminate all pit trades from what should only be the Globex pricing stream for NYMEX/COMEX.

2. Review eSignal's system architecture for tick servers and history servers. Data is different for different servers too often.

3. Address the problem you're having with inconsistent price corrections/filters at various tiers - tick server, history server, eSignal client software (tick/bar caches and all the different chart types and periods).

Unfortunately, tick correction must be done at each tier. Because of this it's easy to have bugs in one tier, which I bet is what you're seeing. It's likely the code to do price corrections/filtering is different for second and minute charts, with one having a bug in it.

[shortandlong]

Clearpicks,

Turns out eSignal has already implemented my suggestion #1.

Appending '=1' (Globex session only) to any NYMEX or COMEX symbol has completely resolved the issue of bad ticks for me. For example, request 'CL G8=1' instead of 'CL G8'.

More info here.

[clearpicks]

Why does the volum of CL G8=1 look so weird?

Originally posted by shortandlong
Clearpicks,

Turns out eSignal has already implemented my suggestion #1.

Appending '=1' (Globex session only) to any NYMEX or COMEX symbol has completely resolved the issue of bad ticks for me. For example, request 'CL G8=1' instead of 'CL G8'.

More info here.

[shortandlong]

I don't use volume, but I get the same thing on daily charts - daily volume for CL G8=1 is missing or wrong (usually too low) for many dates, while CL G8 is ok.

Strangely, intraday volume seems to be ok...