importing data

[treblewide]

I wonder if anyone can help me. I have recently obtained back data for ES for the last 5 years (5 and 30 min data)...now i want to import it into e-signal to run my efs scripts in order to get a proper back test on my systems...however after speaking to some technical dudes at e-signal I have been told it is impossible!!

surely not..there must be a way.....or do i have to bin everything i have done with e-signal and all my efs and start again with Wealth Labs or Trade Station...this would be a nightmare..any ideas??

[gspiker]

Hi,

I have wrestled with this concept myself. IHMO this is one of the things in eSignal that I think is the most frustrating. They don't offer a lot of historical data, and yet make it impossible to use your own should you have it.

The short answer is that they are correct, it isn't possible to do this.

The longer answer is...but if all you are doing is running your own EFS that doesn't use Builtin's etc you could work around this. It would be long and painful, but you could. You would have write code to:

1) Determine when eSignal data ends, so you then switch to your own methods for grabbing data from you historical data file.

2) If it is tick level data, you would have to have routines to handle NEWBAR etc..etc..

3) Do all your own backtesting stats...since the eSignal backtester will not help here.

And even if you did all of this, you couldn't chart your results in any meaningful way (except for the eSignal data portion).

If eSignal would open up their communication API, I would be willing to write a routine that would read OHLC and V info and pump it into the data manager. My guess is it wouldn't even be that hard (but not knowing the restrictions put in place by the DM, can't say for sure).

If they only did this, it would be a HUGE improvement over what we have now. I recently backtested a simple system over the eSignal intraday historical data period that looked very good. Once I moved it to test it on a different platform with more historical data I found some major problems. I'm sure this is not unique.

We just need a way to get more intraday historical data into eSignal... Not allowing this is a huge disadvantage to the users.

Garth

[DuaneG]

Treblewide,

Thanks for the post. The information that you received from the technician is correct. Our core business has been and continues to be as a data provider. We have structured our business to encompass the traders who wish to have both data and software in one inclusive platform. It's for this reason that outside data providers are not interchangeable with the eSignal software. As eSignal is structured to work with a dedicated feed, it is not possible to incorporate outside data into the software.

[treblewide]

Duane,

ok....it is the one and only thing that lets e-signal down..i can understand the company's point of view however it seems really silly to me that i have the data, and you make it impossible to import it...... like Garth we have developed systems that work well in the 6 months worth of tick data you supply but not over a longer period of time.

we run multiple copies of e-signal which I am afraid will be getting cancelled if we find no way around this.

if someone has saved previous data from e-signal were to send it to us...i am sure we could open it and manipulate it?

either that or is spoofing your webserver an option?

T

[gspiker]

if someone has saved previous data from e-signal were to send it to us...i am sure we could open it and manipulate it?

This will not work either (except for maybe tick data...I haven't explored the limits of how much saved tick data can be imported but I would have to say that loading even a few months of tick data into the tick reply would be VERY slow).

either that or is spoofing your webserver an option?

If you were clever, and used a packet sniffer to determine what the data stream looks like that eSignal sends, and then wrote a program to reproduce this, and if you could spoof the Data Managers to connect to one of your servers instead of an eSignal server, and if you could somehow figure out how their authorization system works...well this might work.

Of course it may well violate your licensing agreement with eSignal to do so. I haven't looked closely at it so I don't know.

But that is a lot of work...and I'm not even sure all of it is doable (the authorization system for example may be impossible to replicate).

eSignal has stated their intention to grow the amount of intraday data they keep - but as of the last post on this subject it sounded as if it was a long ways off.

Garth

[treblewide]

it seems stupid to have to go to this trouble just to back test stuff however it would take more work to rewrite a pile of efs studies and get to grips with wealth labs than to come up with a "work around" on the data import...madness!

[RobMc]

Duane

I read your post with interest having come across this problem.

While I understand your point with regard to being a data provider and current data, I don't understand from your post why this would stop you enabling people to import old data as a separate issue from current data provision.

I'm sure there are very easy ways to make sure that your position as a "current data provider" is not compromised, by not allowing data younger than 6m or not allowing outside realtime data feeds etc. The addition of a simple import routine or the ability of eSignal to recognise a locally held file would make Esig awesome imo. To apply Esig charts and efs to a decent chunk of historical data would be amazing.

Your reply was very to the point but can you see that this would be a huge improvement to esig as a total trading package. Or is the answer, bluntly, "tough, go elsewhere if thats what you want 'cos its not what we do nor intend to do in the future". Not being funny but would just like to know exactly where I stand with this important issue and if that is the answer then ok and thanks for being honest :-)

Would really appreciate a reply.

Regards
Rob

[DuaneG]

Hi Rob,

Thanks for the comments. My reply to the post in question was not intended to be curt or short in any way, but rather eliminate any kind of double talk or gray area that someone may interpret. I completely see why this functionality would be useful and desired, but at the same time, I have to articulate the current position of the request. I don't have any idea if the ability to use outside data sources will ever exist, but until then, we don't want to provide any false hopes that it will happen anytime soon or if at all. I hope this clears things up a bit. Thanks again for the comments.

[RobMc]

Duane,

Its a pity, but in any case thanks for replying in such an unequivocal manner.

Regards
Rob

[AndyS]

Hi RobMc,

If I may, I'll try to expand. Right now the software is on our download page and anyone can download, beit a client or not. In a sense, it's free. However, it doesn't work without our data. So, in a sense, it's not free. If right now the possibility exist that a person could download our software and feed it with someone else's data, then truly it is free to them and we receive no compensation for developing & providing the software. I don't think any company will last long on the business model.

In order to do what you request, we need to implement a system by which we can sell the software. This is under consideration and may happen at some time in the future, but will require a retooling of our systems to facilitate that. Like any other feature request, it needs to be prioritized based on the overall needs of our clients and fit into our business model to ensures we can continue to operate as a business.

This has been coming up more and more, and as we hear the feedback, we continue to look at our business model and adapt to the needs of our clients. I appreciate you taking the time to post here with your needs for us to consider.

Thanks,

[Anson]

Originally posted by AndyS
Hi RobMc,

In order to do what you request, we need to implement a system by which we can sell the software. This is under consideration and may happen at some time in the future, but will require a retooling of our systems to facilitate that. Like any other feature request, it needs to be prioritized based on the overall needs of our clients and fit into our business model to ensures we can continue to operate as a business.

Hi Andys,

I'm not sure how the business model works but isn't it possible to allow only users of your datafeed to download the eSignal software?
Make it do some sort of online check as well before it will allow import of data. Only 1 program can be run per license and so only the valid user can import data and he's also a paying subscriber to your datafeed?

Of course, I could be missing something somewhere.

[gspiker]

I'm not sure how the business model works but isn't it possible to allow only users of your datafeed to download the eSignal software?

The problem there is someone could sign up for a month (even the "free" trial month) and then have the software and use anyones datafeed.

However in theory they could (as was suggested earlier in this thread) make it so that only their live data would work with their software but that historical data in a certain format would work no matter who supplied it. In some ways this would be a nice marketing tool - since it would allow people to play with the software for longer than a month to evaluate it (and if you are trying to eval the scripting language for example - more than a month may be needed). It would also solve the problem of allowing users to get enough backtest data and eSignals worry of people using their software to make money without eSignal getting any of that money.

As an alternative they could also allow eSignal users to load data onto their disks...I would be more than happy to buy an extra 80-120 gig disk and fill it with data. After a year or so of doing this I would then have a reasonable amount of backtest data. The data could easily be encoded so I couldn't use it with other vendors software and eSignal could be coded so it would only read the specially encoded data.

Or, as I have suggested before - create a backtest server farm. Make it slow with lots of data an no way to use it for realtime data. Charge me for using it. As long as I can backtest years and years of data it will be a huge win. And eSignal can make money off of it.

I know there is nothing so bad as a backseat product manager (which is what I am playing here)...but there is pain in the user community and we could all benefit from creative solutions to this issue.

Garth

[ykk]

I think the most simple solution for e-signal is to provide a longer period of historical data.
Most people who want to import data is for longer backtesting, so if we can download it from e-signal, importing from outside wouldn't be so important.

[RobMc]

Morning all,

I thought that Duane had put this to bed but from what AndyS has explained, all eSig need to do is put a line in the software that says :-

"Is this person a current live feed subscriber"
If "YES", then proceed
If "No" then self destruct in 5 seconds :-)

Surely something like that is REALLY EASY just as part of some kind of account check which presumably happens every time you switch on your machine and load eSig. It is nothing to do with selling software or people using an outside data source. You just make sure that they are a current user before they can use data from the import facility. And I don't mean current data with anything that entails in terms of managing a realtime stream from outside, I just mean importing a big chunk of historical data. If you're not a current data subscriber then the facility is locked just like certain things are disabled in a demo version. Or, as gspiker suggested, it could be a paid option like all the different exchanges around the world only in this case you tick the box and pay for access to the eSig backtest database.

I would have thought that this should be so important to making eSignal the complete package that if these are the only problems I can't see why it hasn't been done already. Maybe it just isn't that high up on the list of priorities?

Regards
Rob

[TraderJO]

I don't know about the rest of you guys, but the Advanced Charting features of eSignal aren't free for me. They come at a cost of $45/month on your subscription fees.

Correct me if I'm wrong, please.
__
JO