How much historical Tick data will be available?

[mstennicke]

I was wondering whether eSignal is planning to supply more than 10 days of historical tick data. If yes, when and how much data would be available? Also what would be the way of receiving it?

In the eSignal History Data document there's a development note saying: "The goal is eventually supply at least two years of intraday data and grow database even further after that. The project has been in the works for well over a year and will take a major step forwards in late 2005 with the release of our new tick server machines. We hope to be able to offer more that 120 days of intraday history by either late 2005 or early-mid 2006".

Is this is true, when will we be able to get more than 10 days of tick data?

It looks like eSignal builds before 722 were able to pull more than 10 days of TS data, but builds since 722 had limited it to 10 days as a way to prevent out-of-memory crashes!!!? So I am still using the old version of eSignal (Build 719). Is there way for me to know the maximum available days of data for TS data? A lot of times eSignal crashes while I am receiving around 30 days of T&S data for certain instruments without any reason (there's enough memory), and I was wondering if this can be prevented.
For example, could eSignal provide different API functions to return number of available days, or to get data only a for a day X days ago or for a specified period? The way APIs work right now, if I need a TS data for a period 50 days ago, I have to receive all the data before that, even if I don't need it. It's time and resources consuming. Same applies to other intraday data. With the existing method of providing historical intraday data, there's would be no normal machine being able to receive more than 120 days of data - it would run out of memory.

Many thanks,
Mike

[JayF]

Mike,

Thanks for posting this inquiry. Before I answer your questions, I would like to clarify the way eSignal handles historical data. Our servers currently divide the historical data into 3 different categories:

Tick Data - This data is requested when Time & Sales data is needed, but is also used to create tick charts and intraday charts with tick-based intervals (i.e. 390T, 5000V, 30S). We currently carry 10 days of tick data.

Intraday Data - This data is used to fill out minute-based interval charts (i.e. 5-min, 15-min, etc.). 120 days of Intraday Data is currently available.

Daily Data - As you like guessed this data is used for daily charts, but is also used to put together Weekly and Monthly charts. The amount of data in this section varies, but for the most part 10 years is available.

Now to answer your inquires. There are no plans on increasing the amount of tick data beyond 10 days in eSignal. The sheer amount of storage/bandwidth required for this amount is immense.

Intraday data is a different story altogether, and that is what the quote you gave refers to. We are in process of re-writing our "Tick Servers" from the ground up. These servers store and deliver tick and intraday data. This upcoming new version has been written in such a way as to support many different "features" that have been needed for some time. This includes handling splits and corporate actions better and many other things, but will also allow us the ability to begin capturing more than 120 days of intraday data.

Our original target was indeed Q4 of '05 - Q1 of '06, however due to the complexities involved, the amount of testing required, and other business requirements, this is looking to be pushed out to Q3 of this year.

[georger]

I was wondering, since you only provide about 10 days of 'Tick Data', what happens to the data further away in time? Are the servers 'flushed' of information, or, is the information off-loaded to a long term storage area? If that is the case, is there any way to gain access to that data?