symbols not releasing

[mk1979]

Hi there,

I have a question about ReleaseSymbol.

I have an application that initially looks at about 500 symbols and releases many of them depending on certain criteria. The problem though is that esignal still shows almost all 500 symbols as being active and my system performance really starts to degrade.

Here are a couple of scenarios that seem work and not work.


Scenario #1 (which seems to work correctly):

I request 500 symbols using RequestSymbol(symbol, 1) outside of active market hours and release all but 50 symbols using ReleaseSymbol(symbol). Since the market isn't active, I don't receive updates for any of my symbols which is expected and esignal (the data manager) shows that I have 50 symbols still active, which is correct. When I shutdown my app, I release the remaining symbols and esignal shows that there are no more symbols being used. Good so far.


Scenario #2 (doesn't seem to work):

I request 500 sybmols during active market hours and release all but 50 symbols again, but this time esignal continues to show all 500 symbols as being active. This is not expected.

Is the API/esignal having a hard time or missing events when I release symbols due to the processing of my 50 outstanding (unreleased) symbols in real-time?


Scenario #3 (seems to work):

I request 500 symbols (during active market hours again), receive the update event and Release the symbol right away... I do this for all symbols. After all symbols have been released, esignal shows that no symbols are currently active, which would be correct. The only difference between this and scenario #2 is that in #2 I kept some symbols around for real-time processing, here I am not.


So why would my symbols in scenario #2 not be released?

Thanks

[scheel]

symbols not releasing

Hi, mk1979

I have been doing similar things, but have not experienced your particular symbol release problem. I don't understand why you'd still have 500 in your queue unless the release code was somehow skipped because of some switch you have going.

Anyway, I have symbols are being +/- on-the-fly, and latency can easily cause one to exceed the number of allowed symbols. I keep track of the net number of symbols requested and released. I have a sleep state whenever I am about to request a new symbol and the net count has approached my symbol limit.

[Cone Killer]

I suspect your problem is related to latency which I have encounted even when releaseing all symbols. My practice is to program in a pause (i.e. sleep) before proceeding.

Also, as I reported earlier, I check for invalid "0" handles and then then continue to hit eSignal until I get a valid handle.

The message is that eSignal operates on its own mysterious time frame so our code needs to allow for that.

[mk1979]

Originally posted by Cone Killer

Also, as I reported earlier, I check for invalid "0" handles and then then continue to hit eSignal until I get a valid handle.

I'm just requesting symbols at the moment (so no handles involved), but I will try adding some sleeps to hopefully get esignal to keep up with my stuff.

I'll report back with my findings.

[mk1979]

Well, with the right amount of sleeps and trying to keep gui updates to a minimum, I got my symbol count to go down, but with real-time data coming in for even 20 symbols (which updates a simple gui grid), the esignal app doesn't respond much, if at all.

My current workaround is to not request streaming data for symbols, but use a timer to request a group of symbols every 10 seconds or so (non-streaming), but even during the 10 second intervals, the esignal gui can be non-responsive for a very brief moment.

Is this expected when dealing with more than a handful of symbols?

Thanks

[scheel]

I found a similar problem with Excel GUI. I'm dealing with about 1k symbols. And, like your app, I hit the list on a timed basis rather then RTD. The events are coming at you so fast, nothing else gets much of a chance to do anything.

In VBA, I put a DoEvents in the OnQuotesChanged handler. This pretty much solved the problem in Excel. It seems to work for eSignal too. But, in both cases, there is a degradation in GUI responsiveness.

[Cone Killer]

Scheel,
I am an old card punching programmer who doesn't claim to be a wizard on Microsoft's latest Vista technology. However, putting a DoEvents in an event handler sounds a little risky. Seems like a rapid stream of events might blow out your stack with unpredictable results - not a good thing if you have serious money on the line. My inelegant approach (I also use Excel VBA) is to streamline the eSignal event handling and do the analysis in a different process.
Best of luck.

[scheel]

Yeah, Cone Killer, you're right about the possibility of stack overflow. If the DoEvents is likely to burn ones $cache by halting execution, then in the handler I suppose the desired approach is to cache the data into a "thread" that does the processing with an OnTime exit from the event handler. That should eliminate the need for a risky-stack-overflower DoEvents in the handler because of GUI problems. I'll have to get around to doing this asap.
Thanks.

[JimBecker17]

Couple comments. The first is there is a call:

(*m_piHooks)->ClearSymbolCache()

which will clear out released symbols. call this periodically.

eSignal does get confused though and sometimes does not release.

it also does get wedged and has to be restarted. best to do this by walking the task list and killing off winsig and/or winros processes. i would not assume you can wake up eSignal when it goes to sleep.

In my COM handler that catches errors tries to determine this condition and if so kills the processes, resets requests and queues eSignal restart.

when you are processing messages best to keep a recursion flag and not re-enter the routine. i would not call anything that might process other messages. at the same time you might have a timer which processes requests in a polling fashion as well as by message notification.

generally it is best to be ready to manhandle eSignal. there are non-robust features of the API approach and expect to reset and retry periodically. one can make it function, but with tough love rather than gentle nudging...

-Jim

[scheel]

This is, well, a bit discouraging! AveryH wrote: "The Desktop API does not have any type of functionality or feature that would monitor limits or the amount of data being returned." And, now drops the other shoe from JimBecker17, "...you might have a timer which processes requests in a polling fashion as well as by message notification."

So, events are unreliable and polling may work, but be prepared to torpedo the esignal process.

I agree with all of this and only can add that even ClearSymbolCache doesn't necessarily guarantee anything at least with respect to history requests...I've still had OnBarsReceived and OnBarsChanged events arrive after the clear. (I suspect, however, the ClearSymbolCache is working better with RTD quotes.)

I've ratcheted history requests back to half throttle relative to the symbol limit, and even with careful monitoring and use of both ReleaseHistory and ReleaseSymbol, Data Manager still is throwing badboy messages at what I believe is half of the available symbols.

Bottom line...neither ReleaseHistory nor ReleaseSymbol work synchronously. I'm beginning to question whether they work at all.

Hmmm, does eSignal have any API that works?

Thanks for all the posts. It looks like I'm not the only one swimming upstream!

[Cone Killer]

For what it's worth, I am satisfied with the price/performance of the API. It is a relatively affordable research and development tool that can support certain trades but we have to recognize that the API is something of a "hack" that allows developers with limited funding to piggy back on an eSignal consumer service. It is a bit awkward to work with. You have to allow for latency and write your own validation logic. It is a far cry from the hardwired Crays located across the river from the NYSE (to minmize speed of light latency) or even lessor commercial access that eSignal can provide. On the other hand, the API is one heck of a lot cheaper for those of us who need to control our burn rates until Goldman Sachs calls. Best of luck.

[scheel]

Cone Killer placed the right spin on this. And, others too, when they emphasis latency. It is, in my judgment, absolutely essential to allow some sleep time between history requests. The question is how much, and can it vary.

This is a lengthy post. My apologies for that, but there are a few points I've not seen mentioned that bear on all this business of keeping from breaking the Data-Manager-symbols-back. That's when you get the badboy messages that say you'll not get until you pay.

Consider:

(1) You will get an OnBarsChanged before the return from RequestHistory. Yes, your eyes didn't deceive you...you'll likely get an event before return from the statement that produces the event!
(2) You can use OnBarsChanged to throttle latency.

Here's the code, but it's heavily tied into other stuff...it's the forest we're' interested in. Bottom line: we can use the OnBarsChanged event to throttle eSignal's receipt of new requests. This throttling is done relative to eSignal's own ability to handle requests without exceeding symbol limits. I chose an exponentially smoothed throttle that feeds more requests as eSignal increasingly is able to fire them (as measured by hits to OnBarsChanged). Think of it as a maximum latency that is allowed to drift down (and up). This has worked well. I've now processed lists of 2K using this mechanism with a 200 symbol limit. The throttling is the sleep time between consecutive RequestHistory calls. New latency=Old latency*(1-a)+(time-between-hits-to-OnBarsChange)*a. My a is about 0.1 and the maximum latency (default) is 0.05 seconds. So, if you insert a sleep of "New latency" between requests, this is likely to work. Hot dog! There is a kludge that will work. And, like the astute remarks of ConeKiller and others: it's all in the latency. My read on it is that's right...start thinking of the eSignal active-x control as a pooling device instead of an asynchronous one. Odds are, you'll be more successful if you do.

Private Sub eSignal_OnBarsChanged(ByVal lHandle As Long)
Dim dtChange!
'globals can get reset during debug, etc.
If dtLastChange <= 0! Then dtLastChange = Timer() - nThrottleDefault
If nThrottle <= 0# Then nThrottle = nThrottleDefault
dtChange = Timer()
nThrottle = nThrottle * (1 - nAlphaThrottle) + _
(dtChange - dtLastChange) * nAlphaThrottle
'Debug.Print "OnBarsChanged", lHandle, nThrottle
If nThrottle <= nThrottleDefault Then
dtLastChange = dtChange
Else
'hang up...force reset
dtLastChange = 0!
nThrottle = nThrottleDefault
End If
If eSignal.IsHistoryReady(lHandle) Then esignal_OnBarsReceived lHandle
End Sub

Be happy.

[JimBecker17]

Well originally plenty of stuff didn't work at all. It took a lot of effort to convince anyone it wasn't "user error". And it was not pretty. This was perhaps four years ago.

The good news is one can get the job done if they do a bit o' work. At same time don't ever expect eSignal will put resources on improving this area, so you have to live with what exists now.

Regarding overflowing eSignal with requests, I keep track of number of outstanding requests and length of time they have been outstanding. So can throttle number at any time back to a low level and if get no response for a couple minutes can assume a request is dead.

My needs might differ though, I am working with five minute bars and not expecting realtime quotes or anything.

Sometimes I think my side of the world has more logic to deal with their side of the world than the actual implentation...

-Jim