Why does ES trade inside the spread?

[Dave180]

Looking at an ES tick-replay download file I notice that there are instances when there are trades inside the spread - the spread is normally one tick, sometimes two. When it is two there are sometimes trades that occur at the middle price.

As the ES trades on a single exchange there is a single source of quotes and trades - no room for errors of consolidation.

I can understand that the spread may widen, esecially in these volatile times, but how can it be that there are trades inside the spread?

[Dave180]

Answer 1:
eSignal loses some quotes - well, having downloaded some data from CQG I see quotes in their tick data that are not in eSignal's, eg:

Q,081209,155903,913.75,0,10,0,, ; 10B
Q,081209,155903,0,914,0,114,,
Q,081209,155903,0,914,0,113,,
Q,081209,155903,0,914,0,114,,
Q,081209,155903,0,914,0,116,,
Q,081209,155903,0,914,0,118,,
Q,081209,155903,0,914,0,119,,
Q,081209,155903,0,914,0,117,,
Q,081209,155903,0,914,0,116,,
Q,081209,155903,0,914,0,114,,
Q,081209,155903,0,914,0,115,,
Q,081209,155903,0,914,0,114,,
Q,081209,155903,0,914,0,113,,
T,081209,155903,913.75,1,
Q,081209,155903,0,914,0,111,,
Q,081209,155903,0,914,0,109,,
T,081209,155903,913.75,1,
T,081209,155903,913.75,1,
T,081209,155903,913.75,1,
Q,081209,155903,0,914,0,110,,
Q,081209,155903,0,914,0,112,,
Q,081209,155903,0,914,0,111,,
T,081209,155903,913.75,1,
T,081209,155903,913.75,3,
T,081209,155903,913.75,2,  ; Bid exhausted  (10 - 5 x 1 - 3 - 2 = 0)
Q,081209,155903,913.5,0,211,0,, ; Bid quotes lower
                                                       ; No ask quote from eSignal, but CQG tick download has 5 offered at 375
T,081209,155903,913.75,5,    ; In CQG this trade is at ask, in eSignal it is inside.
Q,081209,155903,0,914,0,110,,
Q,081209,155903,0,914,0,111,,
Q,081209,155903,913.5,0,205,0,,
Q,081209,155903,0,914,0,116,,
Q,081209,155903,913.5,0,210,0,,
Q,081209,155903,0,914,0,117,,
T,081209,155903,913.5,3,  ; 2tick spread, trades at bid
T,081209,155903,913.5,7,
T,081209,155903,913.5,1,
T,081209,155903,913.5,1,
T,081209,155903,913.5,1,
T,081209,155903,913.5,1,
T,081209,155903,913.5,1,
T,081209,155903,913.5,1,
T,081209,155903,913.5,1,
T,081209,155903,913.5,1,
T,081209,155903,913.5,1,
T,081209,155903,913.5,1,
T,081209,155903,913.5,1,
T,081209,155903,913.5,1,
T,081209,155903,913.5,1,
Q,081209,155903,913.5,0,187,0,,
Q,081209,155903,913.5,0,171,0,,
Q,081209,155903,0,913.75,0,1,, ; offers come back 

BUT I see CQG still have trades inside the spread, just not as many. So we have two questions:

eSignal - where are the missing quotes please?

Anyone - how can the ES appear to trade inside the spread (assuming CQG hasn't also lost quotes)?

[demarcog]

Dave180,

You can check the CME Groups web site for the specs of their electronic order matching algorithm, as every exchange is different and getting the exact details of any matching engine is not easy, but it is not unusual for trades to occur between the bid and offer for electronic exchanges.

If "market" orders are entered by a buyer and seller simultaneously depending on the exact timing of the order entry frequently a trade will occur "between" the spread.

You apparently have put alot of effort into anaylzing bid and ask data and I was wondering if you found this information useful in actual trading?

Glen

[ScottJ]

Dave,

We moved to a new feed from the CME recently and in some scenarios, there's a conflict with single-side b/a updates. We can temporarily fix the problem by removing a network entitlement from your account. You can PM me or contact service directly to have that bit removed.

Engineering work is in process to resolve for the long term.

Thanks.

[Dave180]

Scott, thanks, could you just explain this a bit more - is there a disadvantage to removing "a network entitlement"?
And does it have a specific name so I end up with the right fix if I contact CS directly?
And can you be a bit more specific about the actual fault scenario. What might get lost? I assume it also affects downloaded tick relay data? When did the feed change?

BTW, have you seen my post about a more serious problem with Tick Replay (only one side is seen).

Also, I see I wasn't the only one to get bad data yesterday, post , is this related?

Glen, Thanks for that insight. I always assumed market orders would match against existing limits, not quite sure I can grasp the logic of matching both sides as market orders, I'll contemplate that.

As for the analysis, well, I've always had a bit of an interest in bid ask analysis. I spent some time looking a Maket Delta's footprints in the context of looking at turns at previous volume and the development of a bid ask bias over minutes and hours in the trading days; they never seemed to work for me and I think that the reasons they are proposed may be less valid nowadays. I still think there is something to it, but my current analysis is slightly different, and incomplete, and hindered by too much inconclusive data!


Dave

[Dave180]

Scott, Perhaps I should add that even in the tick replay file there are missing quotes (and not just the ones dropped by the replay code in eSignal), but in the actual file.

So that makes me wonder whether this "network entitlement" thing is really the issue?

Having analysed a whole day of CQG data and compared it to eSignal (both downloaded toick files) I find that you have 15x as many instances of trades ocuring in inside the spread (caused by missed quotes) affecting 20x as much volume.

Dave

[ScottJ]

The bit I'm referring to is related to outbound conflation, where some b/a data is filtered to reduce congestion to the desktop. That algorithm had been working great until the new CME feed cut-over. The behavioral changes in the new feed require some adjustment to that methodology and those are in process. In the meantime, removing the bit from your account will resolve the issue but also increase the amount of traffic to your desktop, so please keep that in mind as well.

Thanks.

[Dave180]

Scott, Thanks, but did you see the post where I mention that the quotes are missing from the tick replay downloads?

As I see the quotes missing from both the live data and the tick replay downloads I don't see this is a network congestion issue?

Dave

[ScottJ]

If the conflation bit is on, some b/a data will definitely be filtered before reaching your desktop. If that bit is off, all data will be sent to you.

In terms of the tick servers, we have two types: 1)Trades and Quotes and 2)Trades only. Another entitlement determines which one a user goes to but the default is option 2) trades only.

Thanks.

[Dave180]

Scott, we seem to be at cross-purposes here. I speciically said "tick replay downloads". I have previously been told that these always deliver the quotes, regardless of the type of tick-server I am entitled to for live tick delivery. And anyway, I am seeing quotes in the download, just not all of them.

So the problem still remains. Why are some, not all, quotes missing in a tick replay download (as as I said before, this is the quote ticks in the download file that are missing, not the effect of the buggy replay logic as per another post).


Dave

[ScottJ]

If you are connecting to a trade-only tick server, you won't get b/a updates in the tick downloader for regular market hours (you'll see pre and post market quotes only). I see on your acct that you requested the bit for trade and quotes to be turned off on 11/14. Do you need that back on now?

Thanks.

[Dave180]

Scott, I'm lost. You say I won't get b/a updates for regular market hours because I had that bit turned off on 11/14.

Before I had it switched off I asked and was told that the tick reply download was not affected, only the historical part of pure tick charts.

But more importantly, the tick file replay I posted near the start of this thread is from market hours (times are UTC) and it shows the b/a updates that you say I shouldn't get. The problem is that some are missing (particularly some that change the price), not all. I've noted one in the extract I included.

Dave

[ScottJ]

It appears you were given some incorrect information. Tick downloader is driven off whatever tick server you are connected to. That's the only place we store ticks so if you are connected to a trades-only tick server, your replay won't have b/a updates. At least this is the way it's "supposed" to work and does per a test I just ran. If you have regular market b/a updates, then we need to troubleshoot to determine what tick server you are hitting.

Best to call into support for this sort of diagnostic.

Also, there are some known issues with tick downloading on ver 10.2 that will be addressed in 10.3. That may be the source of the confusion here. You might want to roll back to 10.1 and test again.



Thanks.

[Dave180]

Scott,

eSignal Statement on effect on bid ask entitlement on tick Data downloads
Here is what shaeed told me by email Nov 5:
"[1] You do not need the Bid/Ask historical routing bit to download tick data."

This came from an email thread in which I had gone to some trouble to be absolutely sure what the consequences of switching off the quotes entitlement would be. Perhaps you could bring him up to date. But this is also yet another example of different people in eSignal providing conflicting information. It wastes the time of us users. Please could you all sing from the same hymn sheet.

More downloads - quotes still present, but not always
Now here is an oddity. With the market closed (end of week) I did two downloads (ES #F, 10 days), using 10.0 to avoid the 10.2 problems, twice - note that even this is no longer as reliable as it used to be, it generally takes two attempts to download a full ten days.

To begin with (start of 2nd Dec) I see both Trade and Quote data, but then near the end of the day it reverts to the Trade only data you expect (but not at te time you suggested below) and remains like that for all the days (3rd to 12th). This is in both downloads. How can one server have a mix of with / without quotes? Assuming I'm not flipped partway through both the downloads at the same point?

Q,081202,000000,814.5,814.75,30,55,,
T,081202,000001,814.75,1,
Q,081202,000001,814.5,814.75,30,54,,
T,081202,000001,814.75,1,
Q,081202,000001,814.5,814.75,30,53,,
...
Q,081202,230536,846.75,847,37,124,,
Q,081202,230540,846.75,847,38,124,,
T,081202,230557,846.75,10,
T,081202,230613,847,2,
T,081202,230630,847,1,
T,081202,230630,846.75,20,
T,081202,230630,846.75,2,
T,081202,230636,846.75,1,
T,081202,230636,846.75,3,
T,081202,230636,846.75,1,
T,081202,230643,846.5,1,
T,081202,230655,846.75,1,
T,081202,230655,846.75,1,
T,081202,230655,846.75,2,


I then used 10.2 to do a download of 1 day, I got quotes. A four-day download (which takes almost 90 minutes) also had quotes.


Downloads - data still not consistent across servers
As I have reported before, there are again significant differences between the pair of downloaded files mentioned above. I see a difference of about 5000 lines in length. A windiff shows what might be between 5000 and 10000 differences in the files (both files contain data not in the other). These differences occur in the part of the download that has quotes and in the part that doesn't (thus pure trade as well a mixed quote and trade data is different).

Assuming there are no losses in a tick download (especially out of market hours), this in itself suggests that there are server-farm problems distributing the data to the servers and that they don't have all the data, not even the trade data?

Presumably this would this also impact the delivery of complete sets of tick data during live streaming? Thus isn't this the issue for your engineers to look at before we get involved in the the effect any additonal quote throttling that you mentioned in your first reply?

I am on vacation from Tuesday untill the New Year. It would be nice if that would bring more reliable data delivery, and ths less time spent analysing and reporting problems.

Dave

[ScottJ]

Dave,

Technically, the quote you posted from Shaheed is accurate. You don't need the b/a routing bit for tick downloading UNLESS you want b/a quotes in that download. Either way, sorry for the confusion.

The results you post are of interest but w/o knowing what server you were connected to, the time to troubleshoot is extensive. We really need that piece of information to effectively tackle and evaluate the issue. We'll do some testing over the holidays to see if we can spot a problem but we'd really appreciate if you could call into the London office when you find these issues so we can determine your connections.

Thanks.