Historical Data

[sheatrader]

I'm new to the Desktop API, so forgive me if this is answered elsewhere. I've looked but I can't find an answer.

I'm having trouble in getting historical data. My OnBarsRecieved event is not being called (Not 100% sure I've setup the callback correctly.) The IsHistoryReady call does not seem to work.

I'm running on Windows XP using VB 2008 Express Edition.

Some calls do work and I am getting some data, but not all the data that I have requested.

Steve

Additional notes: I have gotten the callbacks to work. I'm still getting inconsistent and incomplete data. I'm only getting about 70 to 80 days of intra-day data. I was told that there is 125 days of intra-day data available. I need those extra days. Also, it seems my requests are processed for a while, but then everything just stops.

[AChan]

It appears that you might be requesting data too rapidly. How many symbols are you requesting historical data concurrently? Try to request data one symbol at a time after all data has come back for that particular symbol. If you have more than 35 outstanding historical data requests, subsequent newer requests could be dropped.

As for the amount of historical data we carry, please refer to this KB article.

http://kb.esignalcentral.com/article.asp?article=1203&p=1

On another note, we currently only support up to Visual Studio 2003 (.Net 1.1 framework).

[scheel]

AChan

Pease elaborate on your statement: 'If you have more than 35 outstanding historical data requests, subsequent newer requests could be dropped.'

I assume you are referring to any history requests which have been made using RequestHistory but for which ReleaseHistory lHandle has not been issued.

But, Requests can be in various states of limbo. For example, I guess I could spool, say, 200 requests and not yet have even received any event for 35 (or maybe I got event OnBarsChanged for, say, 35, but can't do much with it).

So,
(a) 35 could be the number for which a handle has been received synchronously regardless of its state.
(b) 35 could refer to those requests for which an event has been raised.

In an extreme, your statement suggests that I should only spool at most 35 requests and then incrementally spool more only when Release History has been issued for any outstanding requests that have been received and processed.

Finally, "could be dropped" isn't the same as "will be dropped." I'd be very interested in knowing the exact circumstances when a request will be dropped.

How does this 35 relate to the number of symbols I've purchased?

Thanks for any light you can shed on dropped requests situation.

[AChan]

This only applies to historical data request and it has nothing to do with real-time streaming data and your actual symbol limit.

If you have a while loop which fires off RequestHistory requests 100 times consecutively without any wait time in between, certain requests could be dropped. Ideally, you should wait for all bars to come back for a symbol before firing another request.
To safeguard abuse to our server farm, there is a 35 outstanding requests requirement for historical data request.

[sheatrader]

I'm only making one request at time. How do I know that the api will not return any more bars? Once the OnBarsChanged or the OnBarsRecieved event has been called with data, I release the history. Is there an event that tells me the retrieval is done?

I looked at the web page, and I should be able to get 120 days of intra-day data for most all stocks I'm interested in. The max I have been able to get is about 115 days.
Steve

[AChan]

There is no specific event as to tell you when all data has arrived for a particular historical data request. For example, if you are requesting 100 five minute bars, your application will need to keep track of how many bars have been returned to you.

[scheel]

AChan, et al.

Yes, it is essential to monitor bars received relative to bars requested. But, that may be easier said-than-done. My approach now is to over-specify the (history) bars requested and then figure out from what I receive (in terms of timestamp) what I really want.

Why should I overspecify the number of bars? I never thought you'd ask. Hmmm. It is my experience that one may need a deity to figure out what bars (quantity) should be requested at any given point in time to adequately cover the time-span one really is interested in receiving.

For example, the number requested/received on a weekend apparently may differ from the proper number to be requested/received on Friday, even after close.

The work-around of overspecifying is easy to work with although it would be really nice if one could specify both a start/stop timestamp rather than a flat number. (Please, AChan, put that on your, uh, "API enhancements request list.")

But, I'm interested in why event OnBarsChanged is raised when there are no bars, i.e., the statement: "If eSignal.IsHistoryReady(lHandle)..." can return FALSE inside of this event. And, perhaps most perplexing to me: nB = eSignal.GetNumBars(lHandle) can/does return nB=0 in this event and in OnBarsReceived.

So, I ask the deity, "Why? Is this your intelligent design at work? Am I missing something?" Of course, I only can surmise what the deity may respond, yet I'm inclined to believe it must be something like: "Dude, events can be raised for no apparent reason...security prices follow a random walk so, uh, why shouldn't program events?" (Damn, the flood of replies to that last random walk bit likely will spam my email!)

I continue to be fascinated by the purpose/usefulness of the event pair OnBarsChanged...OnBarsReceived.

AChan, if you'd please weigh in on the event OnBarsChanged, I'd truly become a believer. Just what is it supposed to be used for?

Thanks

[AChan]

According to chapter 4 of the eSignal Desktop API documentation, the IsHistoryReady function is used to determine if history data is available in the local cache, therefore it is possible that this method can return false in the scope of an onBarsChanged or onBarsReceived event. If you already have the handler for both of these events, you shouldn't need to use the IsHistoryReady function at all. The FillHistory() method as described in our documentation already covers how to handle bars received and bars updates events in general. You can revisit it as needed.

The question you need to ask yourself is, are you only interested in historical data? If so, you can ignore all the onBarsChanged updates and only process the onBarsReceived events and release the handle when you are done processing. If you would like to obtain both historical data and also bar updates in the future, then you need to first process the historical data and then handle update bars according to FillHistory(). Note that in this case, the history handle should not be released.

If in the event that you are indeed getting zero number of bars back in the scope of onBarsChanged or onBarsReceived event, you can simply add this logic to your code to disgard this particular event .

lNumBars = esignal.GetNumBars(lHistoryHandle)
If lNumBars = 0 Then
Exit Sub
End If

[scheel]

AChan, I just reread your post.

What is FillHistory() method? Where is it?

I just searched the browser...nothing there by that name under IESignal references. Nothing by that name that I can find in my antique copy of "Developing with eSignal."

What am I missing? I'm running version 10.

It would be really great if you would post link(s) to currently available documentation for the desktop api such as "Developing with eSignal" that fully describes everything in the API, including FillHistory.

[Guest]

scheel,

I entered the following search term into Google FillHistory site:esignal.com There were several results returned, one of which was a link to the eSignal Developer Reference, eSignal Desktop API Edition in the FileShare.

The FillHistory method is discussed in chapter 4 of the document.

If you need more information than is contained there, I cannot help you with that. However, if you have problems performing the search or finding the link, let me know.

Hope this helps you.

Originally posted by scheel
AChan, I just reread your post.

What is FillHistory() method? Where is it?

I just searched the browser...nothing there by that name under IESignal references. Nothing by that name that I can find in my antique copy of "Developing with eSignal."

What am I missing? I'm running version 10.

It would be really great if you would post link(s) to currently available documentation for the desktop api such as "Developing with eSignal" that fully describes everything in the API, including FillHistory.

[scheel]

Thanks...it never occurred to me that FillHistory was a private function and not part of the desktop API. Duh!

One of your Googled results included this version of a FillHistory function. It first uses the API to see if anything is there:

esignal.GetNumBars(lHistoryHandle)

and if so, then strips off the bars using:

esignal.GetBar



Private Function FillHistory(lHistoryHandle As Long)

' this function handles all bars received and bar updates

sSymbol = strHistoryHandleSymbol(lHistoryHandle)


Dim lNumBars As Long
Dim baritem As IESignal.BarData
Dim lBar As Long
Dim sBar As String

lNumBars = esignal.GetNumBars(lHistoryHandle)
If lNumBars = 0 Then
Exit Function
End If


Open "./date.txt" For Input As #3
Do Until (EOF(3) = True)
Input #3, sDate
sDate = Trim(sDate)
Loop
Close 3

lstHistory.AddItem sDate
strSymbolPath = "../../DATA/INTRADAY/5MIN/" & sDate & "/" & sSymbol
lstHistory.AddItem strSymbolPath

Open strSymbolPath For Output As #2

For lBar = -(lNumBars - 1) To 0
baritem = esignal.GetBar(lHistoryHandle, lBar)
sBar = FormatBarData(lBar, baritem)

Write #2, sBar
Next lBar

Close 2


lstHistory.AddItem sSymbol
FillHistory = 1

End Function

[AChan]

FillHistory() is not a built in function of the eSignal Desktop API. It merely demonstrates how to handle onbarsrecieved and onbarschanged event. The following could be found at Chapter 4 or the Desktop API developer reference guide:

The OnBarsReceived event occurs when the initial data for a history request arrives:
Private Sub esignal_OnBarsReceived(ByVal lHandle As Long)
'lHandle is the handle to the history request that requested this data
ProcessHistory(lHandle)
End Sub
The OnBarsChanged event occurs for subsequent updates to the data for a history request. This can occur for real-time quotes that alter the most recent bar of data:
Private Sub esignal_OnBarsChanged(ByVal lHandle As Long)
'lHandle is the handle to the history request that requested this data
ProcessHistory(lHandle)
End Sub

Private Sub FillHistory()
' This function handles all bars received and bar updates
Dim lNumBars As Long
Dim baritem As IESignal.BarData
Dim lBar As Long
Dim sBar As String

lNumBars = esignal.GetNumBars(lHistoryHandle)
If lNumBars = 0 Then
Exit Sub
End If

If lstHistory.ListCount > 0 And lNumBars = lLastHistoryCount Then
' Update the top bar
baritem = esignal.GetBar(lHistoryHandle, 0)
sBar = FormatBarData(0, baritem)

lstHistory.List(0) = sBar
ElseIf lstHistory.ListCount > 0 And lNumBars = lLastHistoryCount + 1 Then
' Add a new bar
baritem = esignal.GetBar(lHistoryHandle, 0)
sBar = FormatBarData(0, baritem)
lstHistory.AddItem sBar, 0
ElseIf lstHistory.ListCount = 0 Or lNumBars <> lLastHistoryCount + 1 Then
' Refresh entire list box
For lBar = -(lNumBars - 1) To 0
baritem = esignal.GetBar(lHistoryHandle, lBar)
sBar = FormatBarData(lBar, baritem)
lstHistory.AddItem sBar, 0
Next lBar

End If

lLastHistoryCount = lNumBars
End Sub

A copy of the documentation can be found at:

http://share.esignal.com/groupcontents.jsp?folder=&groupid=185

[taxidev]

History request does not work

I just tried a request for 2 days of 10-minute bar data as described in the documentation, but I didn't get anything at all. The method I used was:
eSignal.RequestHistory(symbol,"10",btDAYS,2,-1,-1)

I've been waiting for over an hour - how long does it take to get 2 days of history?

[AChan]

It should take only a couple of seconds to download two days of 10 minute bars. Are you getting onBarsReceived events? We recommend you to restart your application and eSignal and try again.

[taxidev]

Yes, thank you. I figured out what I was not doing, and now I get the history in completion. I used what you told scheel about requesting one symbol at a time, and it worked very well.