TimeSalesData enhancement suggestion.

[JimBecker17]

Hi guys,

I know everyone at eSignal is trying to get 7.7 out, we are all looking forward to it as much as you guys..

Obviously some issues have been found with T&S data, and some are resolved. It seems that there probably aren't that many developers using this yet, as some of these issues are significant.

This is a suggestion on what problem you are going to start hearing about if/when the use of this catches on.

The problem which is comes up is marshalling this much data across the COM interface. There is just too much data to move across this form of interface if one is tracking a number of symbols trading significantly, with history. (Someone back in 2003 suggested sending arrays of data, but even this is too much overhead.)

Let the suggestion be made that the filesystem is used, with a COM request specifying a pathname for a base file or directory. The eSignal application would then deposit info into these files, and alert via messages when files had been updated. Just use the TimeSalesData structures in binary form, but then break down the records into something following <SYMBOL><DATE>[<TIME>].EPF format for the records.

For current day info one could include the hour, and have a bunch of hourly files, for example. Or subhours. You already have the tick file store/playback in eSignal, so this might just refine it..

This isn't to make my life easier. Currently my model has a second program running to collect all this info and keep track of it like I'm suggesting. It's just even when I'm running this program at low priority and suck off 5000 T&S records at a time it's consuming all my CPU resources. Since this is intended to be a _background_ operation for me it is limiting.

The RequestHistory() bar data could be handled in the same manner, but generally there is way less info there so it's much less of a significant issue.

Don't know if you guys run metrics on your software, but I'll bet if you tried to keep track of MSFT, LU, GE, SINA, TASR all at the same time there would be a littl' bottleneck...

If this gets on a task list it can't be more than a day to implement... and would be very helpful.

Thanks for listening. I can beta it if you want someone to beat on it..

-Jim

[chrisjames]

Jim,

i like your suggestion but would like to suggest the TS data stream might be...

<SYMBOL><DATE>[<TIME>]<PRICE><VOLUME><BID><ASK><TYPE><FLAG>.EPF (OR .XML)


cj

[Guest]

The issue deals with out-of-process vs. in-process function calls that the COM object must make. That is why I keep on insisting that the API has innate limitations. There really is nothing we can do to optimize certain parts of the API. We are in the process of developing a middle-tier API that will probably fit your needs nicely. Thanks for your patience.

[JimBecker17]

Robi,

Don't understand. All my request involves is the following:

create connection object.

ask for specific data via a file.

eventually get told file exists via event.

read file.

in/out process function calls, threading models, DCOM routing and any other complex stuff is not involved. don't need XML, HTML or anything else - just the straight binary data, quickly.

it's nice you guys are working on some other API, but frankly it would be better if what you put out worked correctly and with reasonable overhead. don't simply reinvent something new, make what you have put out work correctly.

The past few months have been quite frustrating only in regard to trying to make your software work as advertised. RequestHistory() now works in beta, but T&S has incorrect data. I am also auto-restarting crashed eSignal from my application probably fifty times a day as it randomly crashes.

I've tried to follow the turbofeed approach with eSignal, but no emails get returned or anything else in that regard. It doesn't sound like this is supported. Expecting something new to show up and work correctly sounds like a long shot.

My patience is still there, and I've tried to be gracious and also help other developers in this forum. With this suggestion it's not clear why you would need a different approach, and could just polish and correct what is there.

If you don't have the resources to cleanup and correct what has been designed for ActiveX, not clear how something new would ever come out better. The interface you are presenting is not that complex as it is, it's surprising has not been purified by now.

I'm not trying to be a butt here, hope it doesn't come across that way. Just want to have your logic work correctly (with minimal overhead) and move on with my application. Way too much time has been spent on this by myself already, need to move on.

-Jim

[JimBecker17]

cj,

With my data storage methods the SYMBOL name and the DATE are part of the filename. Thus the binary only needs subordinate info, which is much smaller. My records include only:

m_lTime
m_lType
m_lFlags
m_fPrice
m_lSize
m_eExchange (enum of string)

When storing a lot of data try to minimize redundant info.

You can load/store the entire day of data in one read/write call.

-Jim

[Guest]

I really do appreciate your patience and I am working on getting the T&S issue that was reported today resolved before the final release. You have been of tremendous help and I appreciate it. The functionality that you are looking for is a part of the API that we are currently developing (which rests on top of a more complicated, yet proven API).

Again, the very nature of how this API works causes overhead that simply cannot be taken out or made more efficient. With the Desktop API, we've been chasing down issues to older OSes, which is not what we develop on, so the W2K problem slipped past us. This was corrected today, but the build will not release externally for a few more days. Our main problem with T&S data is the way it is stored in our databases, which has outgrown its architecture.

Turbofeed is a legacy product, which was a result of our aquisition of Comstock, and explains why you did not receive any feedback. Again, I appreciate your patience and we are working on resolving the outstanding issues in the API.

[JimBecker17]

Thanks Robi. Appreciate you looking into the T&S problem before release. Today has been a bit of a bummer thinking that all the work I've put into this has to be shelved because of bad data. It will be nice to see this fixed.

Sorry if I came across somewhat grouchy, hasn't been a great day. The tub overflowed when composing one of these long messages, so have had the house cooking at 85 degrees trying to dry out the sheetrock downstairs...

Problems with various OSes can be pretty tough, or some opaque COM issues. Remember having a DCOM problem running across five computers, three databases on three separate domains to a different state via modem... Turned out to be we needed a patch from Microsoft for this specific OS.... Ugly. And this was a seven figure job not shippable because of a undocumented patch...

At least you don't have to deal with Asian input method editors.. different on every international version of the OS.. very fun to test, particularly when you don't know the Asian languages!


-Jim

[Guest]

I've also been bitten on a project similiar to that - there was an error to a driver to a machine that the entire system was dependent on. I hope your place dried up.