batch export of historical data

[BinaryMan]

I wanted to export all TAQ data to files in the same way that you can select "download tick data" from the eSignal frontend. Can it be done automatically using the eSignal API? There is no way to demo the API, so before I spend $200 setup fee, I want to know if it can do what I want to do. Tech support was unable to tell me if the API can do this, but from the documentation I've looked at, it seems like it should be possible.

[chrisjames]

Re: batch export of historical data

Originally posted by BinaryMan
I wanted to export all TAQ data to files in the same way that you can select "download tick data" from the eSignal frontend. Can it be done automatically using the eSignal API? There is no way to demo the API, so before I spend $200 setup fee, I want to know if it can do what I want to do. Tech support was unable to tell me if the API can do this, but from the documentation I've looked at, it seems like it should be possible.

Yo, B'Man,

I think we have chatted on elitetrader.com... I am EdgeHunter... please look up my latest posts on elitetrader.com concerning the Desktop API and also PM me if you want... Also read extensively through all the Bulletin Posts here over the last 4 months to be diligent... thanks...


cj

[Guest]

Yes, you are able to programmetically retrieve real-time, or tick by tick data, using eSignal's Desktop API. The number of symbols will vary based on the how frequently they are traded, as well as the computer and network connection, but you should be able to download real-time information for around 250 symbols simultaneously.

[BinaryMan]

I am using the EOD data; all I need is the TAQ data (not in real-time); I usually download it in the evening. I currently have to force the interface to auto-download a list of symbols using PostMessage(), but this method is buggy because the filename dialog box sometimes doesn't respond.

All I need is to export TAQ data for a list of symbols just like the front-end command Tools->Tick Replay->Tick Downloader. I find it odd that there is no implementation in the front-end to do this for multiple symbols. Granted, pulling 1 GB of data per day is a lot (1400 symbols), but I do it afterhours so it should not affect anything during your peak times.

Do I absolutely have to pay the extra for the API when I supposedly can download 500 symbols at once? What is the purpose of that if you can only really do a few at a time manually because of the interface limitations? I have found no way via the main interface or the data manager to do it.

Will the new versions of the eSignal front-end be able to do this? Can the API do this effectively (download one while saving another to disk for effeciency, etc.) using Visual Basic 6.0?

Thank you for your response.

[Guest]

I will attempt to answer your questions. First of all the Desktop API only has Time and Sales data, so I don't know if that will work for you. There is no implementation for multiple symbols in eSignal to minimize the number of requests to our servers and is not meant for computational analysis, which is why we offer our API products. I don't understand what you mean about downloading 500 symbols at once, as there is no feature in eSignal that allows that and there are plans to add it. Your symbol limit is for real-time data, as you can clear out your symbols and request new ones for historical data. Yes, the API can retrieve T&S (not T&Q) data for many more symbols simultaneously using VB 6.0.

[BinaryMan]

>> I will attempt to answer your questions. First of all the Desktop API only has Time and Sales data, so I don't know if that will work for you. <<

It is only necessary to know how the trade was priced in relation to the bid/ask. I am doing Up/Down volume analysis.

>> There is no implementation for multiple symbols in eSignal to minimize the number of requests to our servers and is not meant for computational analysis, which is why we offer our API products. I don't understand what you mean about downloading 500 symbols at once, as there is no feature in eSignal that allows that and there are plans to add it. Your symbol limit is for real-time data, as you can clear out your symbols and request new ones for historical data.<<

As I stated, I am not using real-time data. I just need to download the historical tick database in the evening. I have used the eSignal frontend function Tools->Tick Replay->Tick Downloader to do more than one at a time (from multiple windows), so I know it can be done. The problem is doing it for a list of symbols without user intervention.

>>Yes, the API can retrieve T&S (not T&Q) data for many more symbols simultaneously using VB 6.0. <<

I don't need more than a few at a time; just 2-3 will maximize my bandwidth, but it is faster to download data while dumping another symbol to disk.

In case you're unsure, the data I am talking about is the *.EPF files exported from the eSignal frontend, such as this:

; Symbol=A
; Date=09/13/04-09/13/04
Q,040913,050948,21.8,23.5,4,4,PSE ,PSE
Q,040913,051114,21.8,24.66,4,40,PSE ,PSE
Q,040913,053629,21.8,45.42,4,1,PSE ,PSE
Q,040913,054452,21.8,23.66,4,40,PSE ,PSE
Q,040913,054522,21.8,45.42,4,1,PSE ,PSE
Q,040913,063101,22.81,22.88,1,38,NYSE,NYSE
T,040913,063102,22.82,33200,NYSE
T,040913,063103,22.82,300,MSE
T,040913,063103,22.82,600,MSE
T,040913,063103,22.82,800,MSE
T,040913,063103,22.82,100,MSE
etc.

I'm also curious if the API is cabable of downloading this data for a specific date (if I miss a day for instance).

So in the end I want to download the data for each day for my symbol list.

A-040913.EPF
A-040914.EPF
A-040915.EPF
MSFT-040913.EPF
MSFT-040914.EPF
MSFT-040915.EPF
INTC-040913.EPF
INTC-040914.EPF
INTC-040915.EPF

and so on...

[chrisjames]

[Quote][i]Originally posted by BinaryMan
"It is only necessary to know how the trade was priced in relation to the bid/ask. I am doing Up/Down volume analysis."

B'Man -

are you seeing something i am not ?

in a T&S string for a TRADE there is no bid or ask values to compare against...

in a T&S string for a QUOTE there are bid and ask values but the string (of course) is not a trade but a quote...

so how will you know how the trade was priced in comparision/relation to the bid/ask if eSignal T&S trade strings do not include bid ask values...?

color me curious...

cj

[BinaryMan]

>> in a T&S string for a TRADE there is no bid or ask values to compare against...

in a T&S string for a QUOTE there are bid and ask values but the string (of course) is not a trade but a quote...

so how will you know how the trade was priced in comparision/relation to the bid/ask if eSignal T&S trade strings do not include bid ask values...? <<

The real-time stream he is talking about probably functions that way, but I am asking him how to download the history files that I know have TAQ data in them. (*.EPS files). I just want to automate the process (I already can, but that's by sending PostMessage() to the eSignal/child windows, and I'd prefer to bypass the frontend because it will not release the memory from downloading the files, and consequently, runs out of memory).

[JimBecker17]

I also asked for ability to get files of T&S data as an enhancement/improvement in the following thread:

http://forum.esignal.com/showthread.php?s=&threadid=9887

It would make a lot of sense, but first would like to see the data come out correctly..

I'd like to look at the relationship as well of trades vs bid/ask.

It would seem the flags field used on trades could also allow one to totally skip the quotes and get rid of 90% of data if not desired as well. Would be a lot easier on their servers.

Keeping my fingers crossed that T&S historical data will be fixed when the 7.7 release pops up!

-Jim

[chrisjames]

Originally posted by JimBecker17
Keeping my fingers crossed that T&S historical data will be fixed when the 7.7 release pops up!

-Jim [/B]

Yo Jim…

Second, third and fourth that !

Originally posted by BinaryMan

....snip...
The real-time stream he is talking about probably functions that way, but I am asking him how to download the history files that I know have TAQ data in them. (*.EPS files). I just want to automate the process (I already can, but that's by sending PostMessage() to the eSignal/child windows, and I'd prefer to bypass the frontend because it will not release the memory from downloading the files, and consequently, runs out of memory).

Yo B’Man

Color me still curious and confused !!

Below, are your examples of .eps strings – right.
Q,040913,050948,21.8,23.5,4,4,PSE ,PSE
Q,040913,051114,21.8,24.66,4,40,PSE ,PSE
Q,040913,053629,21.8,45.42,4,1,PSE ,PSE
Q,040913,054452,21.8,23.66,4,40,PSE ,PSE
Q,040913,054522,21.8,45.42,4,1,PSE ,PSE
Q,040913,063101,22.81,22.88,1,38,NYSE,NYSE
T,040913,063102,22.82,33200,NYSE
T,040913,063103,22.82,300,MSE
T,040913,063103,22.82,600,MSE
T,040913,063103,22.82,800,MSE
T,040913,063103,22.82,100,MSE

Ok so from the eps Q strings – Quotes Strings
How would you compare the bid/ask to the price since there is NO price in there just bid ask values and bidsize/asksize values

And from the eps T strings – Trade Strings
How would you compare the price to the bid / ask values when there are no bid ask values in that string…

So what am I missing here… ?

cj

[JimBecker17]

Chris,

In the flags that come with each record it characterizes how the trade took place, so i don't believe you have to even bother with the Bid/Ask if you don't want that info. (Why i said they could possibly save 90% bandwidth.)

Someone from eSignal might want to clarify if I'm not correct here.

Over the weekend was reading some oldstuff from Worden on TC2000 though, and he said it used to work looking at uptick/downtick - but not so much anymore. Of course it could be because he wanted to sell ya on his proprietary MoneyStream and Balance of Power indicators, which haven't used but they sound similiar to the things we are looking for....

To decode the flags field on the original question, here is some code i use to print out what is coming through:

if( pT->m_lFlags & tsfTRADEATASK ) sFlags += "TrAsk ";
if( pT->m_lFlags & tsfTRADEATBID ) sFlags += "TrBid ";
if( pT->m_lFlags & tsfTRADEABOVEASK ) sFlags += "TrAsk+ ";
if( pT->m_lFlags & tsfTRADEBELOWBID ) sFlags += "TrBid- ";
if( pT->m_lFlags & tsfTRADEINSIDE ) sFlags += "TrInsd ";
if( pT->m_lFlags & tsfQUOTEBID ) sFlags += "TrQBid ";
if( pT->m_lFlags & tsfQUOTEASK ) sFlags += "TrQAsk ";
if( pT->m_lFlags & tsfFORMTUPTICK ) sFlags += "TrForm+ ";
if( pT->m_lFlags & tsfFORMTDOWNTICK ) sFlags += "TrForm- ";
if( pT->m_lFlags & tsfFORMT ) sFlags += "TrForm ";


Hope this is a hint!

-Jim

[Guest]

I don't know exactly how to respond here, but yes, there have been corrections to T&S functions in the API that will be a part of eSignal 7.7. I plan to post a download something this week.

You can use the TimeSalesFilter to filter out trades and quotes and yes, that is how to determine the trade flags.

[JimBecker17]

Originally posted by eSignal Robi
I don't know exactly how to respond here, but yes, there have been corrections to T&S functions in the API that will be a part of eSignal 7.7. I plan to post a download something this week.

Wonderful! Looking forward to using the T&S data! Thx Robi.

[lorn]

Hello,

Does anyone know if the time and sales export through the advanced charting program contain these flags, or are they solely accesible through the API?

Thanks
Lorn

[Guest]

The flags are programmatically set and only available via the Desktop API. The export feature only provides the text in the window.