Tick data and TimeSales Bottleneck

[kboobito]

Hi,

I'm trying to get the tick data of some heavy volume symbols using the RequestTimeSales.

The download from eSignal is very fast, the problem is trying to get the tick data across the COM interface one element at a time.

So here came the questions:

A) is there a way of getting the tick data all together ?

B) if no luck with A, then is there a way of stoping the incoming of new ticks after all the historic data is complete ? For example unadvising the symbol but without loosing the handle, so we can retrieve the historic data.

C) continuing with my lucky strike, A and B were negative answers, is there another way of getting those ticks faster without jumping to the Standard API ?

D) hey i'm going to a casino after this one : could you give me an idea of what is going to be the monthly fee for the Standard API.

Many thanks for all your help.

Kboobito
~

[StarrM]

Hi Kboobito,

Working with the Desktop API can slow your retrieval down a bit since you are going through the added layer of the eSignal application. If I'm understanding A correctly, you are wanting to make a request for all of your symbols all at once? You really can't. Ideally you want to make a request, process it, make the next request, process it, etc. This saves load on our servers and on your system resources. It would likely slow you down even more if you attempted to bring down all of that data all at once.

Regarding B, can you give me some more info? What is your application doing? Making a history request, then updating tick by tick? There may be some possibilities here, but I'd like to get a better handle on what you're doing.

C...not so much, no For reasons mentioned above you will find that the Desktop is less efficient that the other two APIs.

D...You'd need to contact our Sales department regarding pricing for the Control or Standard APIs. Note, though that those are usually only given to corporate development groups. NDAs and approval are required.

Don't be discouraged though...you've got a possible straight draw on B ;-)

Starr

[kboobito]

Hi Starr,

Many thanks for your prompt reply.

I liked B too :-).

Sorry about wrong explaining alternative A:

- I mean downloading 1 symbol at a time. The idea is only to get all the historic ticks from the moment i connect to eSignal and upto 2 days back, i don't want to receive all the new encoming ticks.

- After requesting a Time and Sales like this:

filter = new IESignal.TimeSalesFilter();
filter.bFilterPrice = 0;
filter.bFilterQuoteExchanges = 0;
filter.bFilterTradeExchanges = 0;
filter.bFilterVolume = 0;
filter.bQuotes = 1;
filter.bTrades = 1;
filter.lNumDays = 2;
filter.sSymbol = "ES #F";
_tsHandle = esignal.get_RequestTimeSales( ref filter );

- I wait to have all the History Complete inside esignal_OnTimeSalesChanged:

if ( esignal.get_IsTimeSalesReady( _tsHandle ) == 1 )
{
_length = esignal.get_GetNumTimeSalesBars( _tsHandle ) +
esignal.get_GetNumTimeSalesRtBars( _tsHandle );
_isHistoryCompleted = true;
}

- And then I have to loop to get all the TimeSalesBars using the GetTimeSalesBar function:

if ( _isHistoryCompleted )
{
n = 0;
while ( n < _length )
{
ndx = -( esignal.get_GetNumTimeSalesBars( _tsHandle ) +
esignal.get_GetNumTimeSalesRtBars( _tsHandle ) - n - 1 );
_ticks.Add ( esignal.get_GetTimeSalesBar( _tsHandle, ndx ) );
n++;
}
esignal.ReleaseTimeSales( _tsHandle );
}

The problem is that getting all those TimeSalesBars one by one is a very slow process. For example receiving all the ticks of 2 days for ES from your servers is less then 1 minute, but getting them through the COM interface so you can use those ticks inside your program with this loop could take 30 minutes ( around 500k ticks ).

A) So is there a way of getting all the TimeSalesBars together, instead of one by one ?

B) If that is not posible, is there a way of unadvising the symbol but without loosing the Handle ? I mean after receiving the Historic TimeSalesBars from your servers, i don't want to continue receiving the new ticks. When you loop to get all the ticks one by one, is slow by itself, but also is slower because it constantly updating encoming new ticks.

C) I'm using Visual Studio 2005, and when doing the loop to get the TimeSalesBars one by one, if i do that during market hours on a symbol like ES or NQ it gives me the following exception:

[System.InvalidCastException] {"Unable to cast COM object of type 'IESignal.HooksClass' to interface type 'IESignal.IHooks'. This operation failed because the QueryInterface call on the COM component for the interface with IID '{C5916B45-A7EB-4919-A742-AD47AE3AE996}' failed with HRESULT: 0x80010100 (System call failed. (Exception from HRESULT: 0x80010100 (RPC_E_SYS_CALL_FAILED)))."} System.InvalidCastException

Could be during any iteration of the loop, tick 53000, 134000, any.

If i'll do the same loop after market hours, with an small tick volume or with a lower volume symbol it works fine.

I try doing the loop inside esignal_OnTimeSalesChanged and also from a different thread but with the same result.

Any idea? I can send you the code.

Many thanks again for all your help.

I'll be back on tuesday.

Great Weekend.
Kboobito
~

[AChan]

Here is the logic for obtaining only history data after requesting timesales ticks, note that FillTS() is called
when there is a OnTimeSalesChanged event. By releasing the
handle after all history bars arrived, you will not have to handle
any realtime bars.

Anthony
eSignal Developer Support

Private Sub FillTS()
Dim lNumBars, lNumRtBars, lBar, lDiff As Long
Dim item As IESignal.TimeSalesData

lNumRtBars = esignal.GetNumTimeSalesRtBars(lTsHandle)
lNumBars = esignal.GetNumTimeSalesBars(lTsHandle)

' When the number of real time bars is zero, we are still
' waiting for all history bars to arrive, ignore this set of bars
' for now
If lNumRtBars = 0 Then
Exit Sub
Else

Debug.Print "History Bars --> " + Str(lNumBars)
Debug.Print "Real-time Bars --> " + Str(lNumRtBars)

' How many history bars do we get in total?
lDiff = lNumBars * -1

For lBar = lDiff + 1 To 0
item = esignal.GetTimeSalesBar(lTsHandle, lBar)
sBar = FormatTS(item)
list.AddItem sBar, 0
Next lBar

' Here, we can release the handle because we have processed
' all history bars for this symbol
esignal.ReleaseTimeSales (lTsHandle)
End If
End Sub

[JimBecker17]

I check back periodically to see if anyone else asks about this functionality. It's probably been a year since the last time so here is the whine...

It has been several years since GenSpoo originally requested this and I have done likewise. The issue is not _understood_ or someone could fix it in a few hours.

The issue exactly is that timesales information comes across the COM link ONE element at a time, with all the overhead. So if collecting T&S data for many symbols probably 95% of the time spent is moving the data FROM WINSIG TO THE APP. It is not downloading the data from the server to the local PC.

The SOLUTION is to return an ARRAY of T&S data in one chunk, or otherwise put it in a FILE or anything other than what is currently done. This is pretty trivial but years go by and nothing..

If anyone at eSignal had written a program where they retrieved historical T&S data over any time frame they would realize all the wasted time because this functionality does not exist.

This is very fixable if someone at eSignal cared enough to break loose this bottleneck.

I retrieve all the ticks after hours, this isn't a realtime issue or anything. Also having N handles open isn't a problem, just moving the data across the COM link.

Anyone interested in more detail just search for posts under my name - the more recent ones were this issue.

-Jim Becker

PS - For those who are still using the API and this functionality, one trick is to make sure that eSignal does not have windows open when you are doing this - it degrades badly based on the number of windows open in the GUI when doing this continual COM requesting.

[sundar]

Hi,

I just started trying out the API. Here is my experience. I tried to load just 1 day of zn t&s data. I get a IsTimeSalesComplete quite soon, about 10s. I wait till I get that and then do GetTimeSalesBar() about 270,000 times to get the data (that's how many got returned). It took 20 minutes! I hope this is some error on my part and not the expected behavior. Seriously, no legitimate api can take 20 minutes to process 1 day of history. I actually used the sample program that someone had converted to C# and modified it slightly to do this. (I did no processing of the data to ensure that I only measure the time taken by the GetTimeSalesBar() function. Also this was done on a saturday, so no realtime congestion issues). Any guidance will be greatly appreciated.

Sundar

[atlas]

Is this still an issue. I am thinking about using the API.

[JimBecker17]

As far as I know this has never been changed. So I have one app that just grabs the T&S data, and my hunch is over 98% of the CPU will be this call or Idle time waiting for the data to stabilize.

My primary app does GetGetBars() and it spends about 22% of it's time fetching bars because they have to come across one at a time.

To put that in context, no other part of my application uses even 2% of CPU.....

If there have been any updates to return arrays please post to this thread. Thanks.

-Jim

[JimBecker17]

Ok - I lied. But didn't know it. Ran the profiler when fetching T&S bars for 25 symbols and the profiler info is below. The combo of getting the bars, finding out how many bars and the message pump is only 97.5%.

(second column below is the % for the functions)

If you note everything I'm doing in my app itself drops off to under 0.1% pretty darn quick....

As stated in a plea oh so long ago, this is like unloading an eighteen wheeler full of twinkies one gleaming gold twinkie at a time......

-Jim




Func Func+Child Hit
Time % Time % Count Function
---------------------------------------------------------
471207.679 84.3 471441.222 84.3 149850 IHooks::GetGetTimeSalesBar(long,long) (esignal.obj)
64720.527 11.6 558762.180 100.0 513 CWinThread::PumpMessage(void) (mfc42d.dll)
8747.252 1.6 8754.570 1.6 3547 IHooks::GetGetNumTimeSalesBars(long) (esignal.obj)
2329.908 0.4 2329.908 0.4 156 CStdioFile::Open(char const *,unsigned int,class CFileException *) (mfc42d.dll)
1465.037 0.3 1465.037 0.3 68692 CStdioFile::ReadString(class CString &) (mfc42d.dll)
882.978 0.2 882.978 0.2 104 CFile::GetStatus(char const *,struct CFileStatus &) (mfc42d.dll)
746.451 0.1 746.451 0.1 165839 CString::Format(char const *,...) (mfc42d.dll)
598.562 0.1 598.562 0.1 370652 operator delete(void *) (mfc42d.dll)