Streaming Trades Tracker and Backtester

[LetUsLearn]

Hello All,
Attached you’ll find 2 notepad files, save the “TradingLibrary” file as a .efsLib file in eSignal’s “FunctionLibrary” folder ("TradingLibrary.efsLib") and the other file (“TradeLibrarySyntaxInMain”) you can save however you want, it just shows you how to use the trading library functions and the global variable object...you could also use it as a shell to test an idea out in an “efs” program. What’s nice about this approach is that you take a lot of the clutter out of the program that you’re trying to use to analyze the markets with. I used to paste about 40 global variables into the top of my program and then pasted a long list of lengthy functions underneath main()…what a pain, recently I put it all into a function library that I decided other people would probably find useful.
Some of the benefits of doing this are:
Your trade results will stream to the formula output window in real-time as they are happening, so you can see how your system is performing in real-time. Those results are automatically saved to an array that’s part of the global variable object, so that you can print them to the formula output window and easily copy them into excel so as to compare the streaming trades with a reload of the program…the back-test results. There can be large differences between the two, for example if you are using the closing price to enter/exit a trade, while streaming it’s the current price, but the program “knows” the future closing price when the program is loading…thereby avoiding a lot of bad trades…knowing a bar’s close before it happens is a huge advantage.

You can customize the trading library and the global variable object into whatever you want, but I’d suggest that you keep it generic so that you can use it for any new ideas that you come up with. Keep program specific code in the program and put generic behind the scenes functions into the library.

The library includes a “PrintValues(…args)” function that’s handy to use when you want to print something to the formula output window, instead of

debugPrintln(getCurrentBarIndex() + “\t” + var1 + “\t” + var2 + “\t” + var3 + “\t” + var4);

Pass a comma separated list: TLib.PrintValues(var1, var2, var3, var4); The bar index is automatically included.

If anybody has any question or finds this useful, let me know. If there's no interest, then there's no purpose posting on this forum.

Sincerely, LetUsLearn

[LetUsLearn]

I used to use the PrintValues function to print the trades before I wrote the PrintTradeValues function, as such it had some needless code in it, like the bar index and other stuff, so I updated it. Also before I put this out there I tweaked few things and introduced an issue, so if you downloaded the first version, just delete that code and replace it with this one.

[demarcog]

Hi LetUsLearn,

Thank you very much for posting this awesome function library. You obviously are a talented, experienced developer and as an IT guy (not a developer but hacked out some code here and there) I've seen enough code to recognize you are world class at that. I appreciate you sharing this with the forum and encourage anyone serious about developing trading systems to take a look.

I installed the efsLib and started playing around with the TradeLibrarySyntaxMain file and have a few questions.

I tried to add a simple entry and exit criteria when price is above below a moving average.

Unfortunately I'm getting a perplexing error on the return statement of all things I included the syntax error message as a comment on line 24 of the attached file.

One question is once you have a back tested strategy do you run it real time using the EFS_AT_EXAMPLE structure discussed in previous posts?

I noticed you use limit and stop orders when back testing which suggests that when run in real time you do not specify setComputeOnClose() in your real time EFS's and run your strategy on each tick?


If so do you use one EFS for back testing and another for real time trading or do you combine the Strategy and Trade methods in the same efs?


I've spent considerable time integrating both Strategy and Trade methods into the same EFS as it's preferable for me to only maintain one efs but it was a real challenge. It involves whether you specify setComputeOnclose() in premain and if so setting a bRealtime flag based on NEWBAR and BarIndex of -1 as without setComputeOnClose() specified the real time bar index is 0.

I then check for backtesting mode by doing the following


if (nBarState == BARSTATE_NEWBAR && !bRealtime && !Strategy.isLong()) {
Strategy.doLong("Long Entry", Strategy.CLOSE, Strategy.THISBAR, nQty );

}

and test for real time based on the bRealtime flag being set when either the barindex was -1 for setComputeOnClose() or 0 when setComputeOnClose() is not executed in premain and barstate is NEWBAR.

if (bRealtime && nBarIndex == -1 && nBarState == BARSTATE_NEWBAR){

buyCallback();

} ...

This technique was developed and published many years ago and I was wondering if you use this or another technique to determine when to execute real time vs Strategy functions?

In terms of plugging an practical trading strategy I've done more back testing then I care to mention and have found all to many indicators/strategies to not show a profit on any time frame and in almost every market with some exceptions.

In volatile time like these on an intraday basis (and I focus exclusively on intraday) I have found the Donchian based strategies found in the BT Examples folders to be good starting points for a trading system and mention it partially because I see you have a default stop setting of the low of a 10 period Donchian channel.. I also recently been working on a ICHMOKU based strategy.finally looking for pullbacks in a well established trending environment.


I've always found the pink elephant in the room on this forum is that rarely if ever is there any discussion of what others have found promising and what to avoid. I have no problem telling the world to not waste time mechanical trading systems based on moving average crossovers, price above or below a moving average in and of themselves. They may have some use for entry but the devil is in the trade management, trailing stop, profit target etc aspect of the system.

It's interesting that even masters of trading systems like Well's Wilder when asked why if he found something profitable he would share it. His answer was that the systems would inevitably be tweaked in terms of parameters, markets, time frames etc . It's interesting that I posted a heads up on the BT_Donchian trading systems in the BT Example folder as in volatile time like this, they make money and offered to collaborate and not a word from anyone. For decades I've jumped on and tested every new indicator/strategy even hinted at in this forum as being profitable and came up short on each one. Apparently my approach and methodologies are terrible wrong as I interpret the lack of interest as others are either not trading or making lot's of money doing so.


At the end of the day we are all essentially I would think looking for a profitable trading system and wondering if you could suggest a direction in that regard?

Thank again very much for all you assistance and apologies for going on and on if you ever want to continue the discussion offline please let me know I'm not sure if I'm using the forum appropriately with a lengthily ramble.

Glen

[email protected]

[LetUsLearn]

Hello Glen,
Although I’m aware of the setComputeOnClose() functionality, I have never used it before, although I don’t see anything wrong with using it if you don’t want your script to run on the current bar. Whenever I’ve wanted that functionality, I’ve always used BARSTATE_NEWBAR to execute my code, or if I was using the close to decide what to do, I would use close(-1) to look at the prior bar.

The Obj.nBuyStop and Obj.nSellStop calculations really shouldn’t even be in the trading library, they are left over from a system that I was working on when I developed the library, and I never took them out…they really don’t belong there. Like I said in my own post on the forum, you shouldn’t put any program specific code into the library, it should be generic so that any programs can use it. It’s good to have those variables as part of the global variable object, but the stop assignments in the GoShortLimit and GoLongLimit functions should be deleted because the stop assignments obviously belong to the programs, they’re not all going to calculate a stop the same way…so delete both stop calculations but keep the variables in the object!

As to whether I “use one EFS for back testing and another for real time trading”…no I do not since there’s no need to, at least not since I put the functionality into the trading library to turn off eSignal’s back-testing code. Not that there’s anything wrong with their back-tester, but I’ve had the back-tester cause my program to crash because I couldn’t go long because I didn’t cover my short or something like that, so if I’m trading with real money, I have a parameter in main() where I can set Obj.bBackTestOn to false so it doesn’t run eSignal’s back-testing code. The brilliance behind this is that the market gets to tell me that I screwed up instead of eSignal, but at least the program keeps running…lol.

On one of my displays, I always have the formula output window up, and as the system that’s running executes closed out trades it shows the statistics for that trade. It will do so in real time and it will do so as the program loads. After you load the program you will see that the numbers from the trading library are the same as the numbers from eSignal’s back-tester, with the caveat that eSignal will also show open trade results that you would have to add to my numbers to get identical results. The benefit is that you get to see your trade results in real time, and if you put a button on your chart to print the aTradesArray, you can analyze the real-time results against what the back-test results are when you reload the program so as to see any weaknesses in your program.

So, where you have “Strategy.doLong”, put “TLib.GoLongLimit(Obj, close(0), true);”. Then wherever your code closes out the trade, instead of “Strategy.doSell” put “TLib.SellLong(Obj, close(0), false, false);” The last Boolean argument is for whether you want an alert sound or not. If Obj.bBackTestOn is assigned to be true, then you’ll have both eSignal’s back-test results and mine from the trading library…that’s before the big IF. And that is IF I didn’t forget to tell you something and you do everything right…lol. ALWAYS have multiple backup copies of your work!!!

The main advantage for you to use the library is that you can watch trades stream in real-time, you don’t have to wait and run a back-test to see them. Also you can compare how your system does in real-time with what the back-test results are. Depending on the code, there can be significant differences which gives you something to investigate in how to make your system preform more like those beautiful back-test results.

Like you, I feel like I wrote a book, so in closing let me say that line 24 was blank on your code, I’m uploading you a picture of what I see along with changes to the code that you uploaded.

Happy coding!
LetUsLearn

[demarcog]

LetUsLearn,

Thank you for the response I need some time to read through it carefully and realized I made a mistake in that the error I'm receiving is actually on line 23 and if you load the efs you should see the following error in the formula output window:

TradeLibrarySyntax.efs, line 23: return not in function
if(nSma == null) return; // Exit the formula if invalid data.

I've gotten very many errors over the years and this is the first time getting this error and kind of at a loss for what it means.

I'm reading though your post and appreciate all the insight you provide unfortunately living in downtown NYC is a bit distracting at this moment as going food shopping right now requires protective gear and waiting on line around the block as they only let in a limited number of people. So the fun or parsing you helpful information will have to wait until later I did want to quickly correct the error I made. Originally I copied the formula out error message referring to the error on line 23 into line 24 of the efs itself, but must not have save that, my bad.

I wanted to show you a bunch of code but didn't want to bore the masses with coding trivia on the forum and thought email might be a better way of communicating.

Have to run but to be continued after I unpack and disinfect many bags of groceries.........thank again

glen

[email protected]

[LetUsLearn]

Glen,

The only code that should be outside of main() and preMain() is other functions, global variables, and prototype statements, as such I don’t understand why you put the code between lines 13 and 23. Your error message is due to having a return statement outside of a function. I would not expect the above code to work even if you deleted the return statement. Another issue is “var nSma = xSMA.getValue(0);”, you can declare nSma as a global variable there, but you can’t assign it to be xSMA.getValue(0) outside of main()…and for that matter nSma should be declared as a local variable in main() with a bInit, BARSTATE_ALLBARS, or whatever statement after which you assigned it to equal xSMA.getValue(0) elsewhere in main() where it’ll update continuously for your program.
So, put all code related to your system inside of main() unless it’s one of the things in the first sentence of this reply…good luck!

[demarcog]

LetUsLern,

That was a careless copy and paste error on my part I copied the code above main and meant to copy in one line lower....duh....apologies.

Thank for pointing that out now the strategy loads.

To confirm I'm understanding and to put things in the proper context here is my overall approach to implementing a strategy.

Years ago before the generic broker function Trade Class library evolved (really only since eSignal published the EFS_AT Example code) all the focus was on the Strategy backtesting methods. As time went on the issue of OK now my strategy tested well and I want to run it real time using the generic broker functions. The question was how to do that. Ideally one script that will run the backtester when bars were loading and the script would when running in real time and switch to the generic broker functions.

This was a bit tricky and involved detecting when all the bars were loaded and the script was running in real time, setting a flag to that effect and from that point on not executing the back tester functions but rather the Trade, broker functions.

I'm obviously not a professional developer but a technical analyst/trader and short of putting all my source management into a git structure I have enough trouble maintaining change/version control with a singe script so wanted to have one script that handled backtesting so I could tweak the strategy and then when satisfied load it up in real time and have it trade automatically.

I've worked for with a phenomenological developer Steve Meyer who I met and unfortunately lost touch on this forum and we together (he did the coding) developer a massive, truly beautiful Trade library that essentially did that, would backtest the strategy and when in real time, switch off to real time trade automatically. The library is somewhat outdated but I'm sure you would appreciate the beauty of this class library and would be happy to share it with you..

When you are satisfied with the performance of a strategy and want to trade it with real money do you write a separate EFS to trade in real time?

Thanks,

Glen

[LetUsLearn]

Hello Glen,

There’s no reason that eSignal’s back-test code, my trade library code tracking trades, and real-time auto-trading can’t be running simultaneously. That being said, I’ll add the one caveat that I said earlier in this forum, and that is that I have a parameter (Obj.bBackTestOn) in main() where I can and do set eSignal’s back-testing to false because it crashed my program before during real-time trading because the market was moving so fast that I didn’t get a fill and I tried to liquidate a position that their back-tester said that I didn’t have…so I was forced to reload the program.
If you look at my trading library, you’ll see that it prints the program’s trades and trade prices…the same trades that the strategy is implementing, so there’s no need to run both eSignal’s back-test code and my trade tracker other than to find out if my trade tracker assumed that there was a fill when in reality real-time market conditions wouldn’t have allowed it to happen. My trade tracker doesn’t try to confirm that the fills really happened, it just assumes they did. If I have made some logic errors about my fills in a new system, eSignal’s back-tester will tell me. Below is an example of some code inside an “if” statement that causes my system to go long:

1. if(bTradingLive != null) BuyWithLimit(getMostRecentAsk() + Obj.nRevisePrice, true);
2. if(isLastBarOnChart()) TLib.GoLongLimit(Obj, getMostRecentAsk(), true, Obj.bTradingLive);
3. else TLib.GoLongLimit(Obj, TLib.GetPrice(true, nTradePrice), true, Obj.bTradingLive);

In line 1, I test to see if I’m connected to live trading, bTradingLive == true if connected for real trades, false if connected for paper trades, and null if there isn’t a brokerage or eSignal paper trade connection. If not null, I have my own real-time trading function place the buy order with getMostRecentAsk() + Obj.nRevisePrice as the limit order. So I’m buying the ask so as to insure a fill, and I have a parameter in main() where I can add to the ask so as to insure that I get a fill. I adjust this adder depending on how volatile the market is trading. If my system says to buy, I basically want to buy at the market but I won’t put a market order in for obvious reasons.

In line 2, I test to see if the program is done loading, in which case I call the functions in my trading library with a limit price of getMostRecentAsk(). The trading library is going to assume that I get filled at that ask price for the purpose of tracking how the system is performing. I don’t add the Obj.nRevisePrice to it because the odds are I will be filled near the ask and adding to it would make the trade results needlessly look worse. I’m always comparing how close the fills are between line 1 and line 2 and they track close enough so as to convince me that my assumptions are sound.

In line 3, the program is still loading so it’s going long at a calculated limit price as determined by the system, since there isn’t going to be an ask price to access.

One of the things that you may notice in the above code is that the call to TLib.GoLongLimit has 4 arguments instead of 3, that’s because it’s using my “TradingLibrary-AT” instead of the “TradingLibrary” that I shared with you. The main difference between the two is that the auto-trading version has a lot more variables added to the global variable object that are needed for live-trading. The main reason for Obj.bTradingLive is to stop the alerts from being triggered in the trading library since I’m having the alerts handled by my live trading functions.

So after a long-winded explanation, let me say that if you are just watching your system trade, let eSignal’s back-test code run along with my library code so that you can see the trades being printed out as they are happening. If you have unrealistic limit order fill expectations, eSignal’s back-tester will point those out for you to fix. Put a button on your chart so as to print the Obj.aTradesArray to the output window, then reload your program so as to see the difference between what happens in the real world and what happens in back-test land…you can then copy and paste the 2 results into excel for comparison. When trading in real-time with real money, have a parameter in main() where you can turn eSignal’s back-tester off, but keep my trade tracking library code running so you can compare your live fills with what my trade tracker is saying that you should be getting.

So in conclusion, the trading library that I provided gives you a real time tracker that can run simultaneously with your real-time system. Everywhere in your system that you go long, just call the TLib.GoLongLimit function, all that it’s going to do is to track your live trades, it doesn’t place any real trades.

Lastly let me say that your concern about the script loading and making the switch from the back-test functions to the broker functions is not necessary, your broker functions aren’t going to do anything while it’s loading, since you can’t trade the past…not that we wouldn’t like to find a way to. Nothing happens after the program is loaded until you enable auto-trading on the chart. While connected to my live trading broker, I have programs running with all of the live trade functionality running except that the chart isn’t enabled for auto-trading, so nothing happens except that it gets null for the fills. Instead I have another chart connected and enabled for auto-trading that just has buttons on it so that I can place the trades from the other system manually, because it has too many bad trades that I haven’t figured out how to program out yet.

If you have a class library of generic code, why not share it on the forum with everyone? Maybe other people will find it useful?

LetUsLearn

[demarcog]

LetUsLearn,

Thank very much for the explanation understood completely. I now see you point and missed it previously and it's well taken that there is no reason to run both the back tester and it is extremely useful to also have the trades reported real time in the formula window. That is a fantastic feature. Your code is amazing very few people have the ability to code as wee as you can.

The trade library I mentioned was written by Steve Meyer and as soon as I can get in touch with him I know he'd be happy to share it as he mentioned making it available.

The issue is there is a copyright statement in the source and I honestly don't know if I I'm allowed to share it with the world in that regard but will research the legality of my doing so.

He also wrote two other tools. One was a windoews DLL in C++ that essentially was a driver of the eSignal back tester. This application would open up, you would supply the efs you wanted to back test, and it would invoke a genetic optimizer to run through all the parameter combinations via back testing runs and report on the most profitable combination. Ignoring the issue of optimization, it was a very useful facility. I've been in IT for 30+ years, immersed in all kinds of code, and Steve like yourself are both amazing developers. I supplied the specs and the more practical trading component as Steve was not a technical analyst or trader but he cranked out some truly amazing code, well documented, consisting of hundreds of concise functions, the program could be 10k lines and it would all be broken down into 5-10 line functions which is pure brilliance and not easy to do. He also wrote amazing documentation and his code was equally self documenting. He wrote a few of eSignals related product manuals, QCHARTS or QLINK not sure which one,

I am attaching the genetic optimizer library. The copyright statement for this seems to release it for educational. I have to look for the DLL and see if it still works on the more recent windows versions, I will redouble my efforts to track him down as we were good friends at one time and lost track of one another as in this day of Open Source and GIT etc, it would be great to collaborate on enjancing eSignal's capabilities. There is Qunatopian, MT4, and many other competing products which are all inferior to eSignal and it's really in all our interest's to enhance eSignal user base.

Just a few questions: When you develop a strategy do you maintain separate version for your trade library and the real time version?

I and I'm sure many others would be very interested and find great benefit in the real time version of your library would you be willing to share it with the forum?

If you also had an example implementation on the use of it obviously without divulging any secret sauce would also be helpful.

Thank you,

Glen

[LetUsLearn]

Hello Glen,

The TradingLibrary_AT is almost exactly the same as the one that I already posted, except that it has a lot more variables in the global variable object. Other than that there’s an if statement in the trading functions that will cause the alerts to not sound off if there’s a live or paper trading connection…that’s about it.

I started the _AT version of the library because I wanted to keep the original version as a backup so that if I made lots of large changes that ended up causing problems, I could look at the original version. In the end I haven’t made any substantial changes to the original version yet other than adding a lot of extra global variables.

There's no need to maintain separate versions of your programs. That being said, they all start out as an idea, and if it shows promise you're going to be backtesting it to see how it performs, and with my TradingLibrary you can watch it trade in real time and compare those results to what reloading the program shows as the backtest. Eventually if it all looks good, your going to put the code in to do live trading.

As for “GeneticOptimizer1_5.efsLib”, I briefly scrolled thought it and didn’t see where it has anything to do with trading, although I noticed a function that wrote to an excel file that I will probably look over some more eventually. A while back I tried out eSignal’s “DDEOutput Object” which returns a dde (dynamic data exchange) value from main that can be streamed into an excel file, eventually I figured I’d see what I could do with it, so maybe your file will help with that.

There’s no need to hassle with trying to post the other files, it’s not necessary.

LetUsLearn

[demarcog]

LetUsLearn,


I found an early version that both me and Steve worked on that was not copyrighted that is very much trade related, it is in two parts as it exceeded the file size limitation of the forum.

There were a lot of enhancement in later version that if anyone else is interested working on it let me know, its 5 times the size and I would have to break up into 10 parts.

You mentioned that _AT version of you library is almost identical I'm assuming the difference is in executing the broker functions rather then the strategy.

This is the version that I and I'm sure other would find helpful.

glen

[LetUsLearn]

Glen,

If you read the first 2 sentences of my prior post you'll notice that there's no reference to code executing broker functions, that's because there's none in it, nonetheless I'm attaching it here. It mostly takes all of the global variables in eSignal's EFSAT_EXAMPLE_2014_V2.02 and moves them into the library so that they're not cluttering up my programs. I prefer all generic global variables that can be used by multiple programs to be in the library, whereas program specific global variables need to be in the program.

Thanks for uploading the TradeFunctions library, it looks like a pretty impressive effort. I'm not sure what I'll do with it yet, but over time I'll spend time studying it over, so far I just scrolled through it and found it to be quite impressive!

LetUsLearn

[demarcog]

LetUsLearn,

When you mentioned an "AT" version I understood that to mean auto trading and that's what you were going to share with the forum.

Appreciate the comments on the Trade library we worked hard on that and it evolved and grew considerable, Steve is an amazing developer.


glen

[LetUsLearn]

Glen,

If I’m going to make substantial changes to a program or a library, I start a new version of it so as to have the old version as a backup, that’s what I did when I created the _AT version of the TradingLibrary which I intended on using for auto-trading. I started out by adding all of the global variables that I would need for auto-trading and then added a parameter to the trade functions to stop the alerts for when I was connected to live or paper trading…that’s as far as I got, and I’m not sure when or if I’ll do anything else with it since for now it serves my purpose for auto-trading.

LetUsLearn

[demarcog]

LetUsLearn,

When you say the back test library servers your purpose for auto trading I don't understand as the library doesn't have any generic broker functions.

Do you do autroding or the stratgies you developed.

The forum has hundreds of examples of back testing programs and the only code I've seen for AT is the EFS_AT efs discussed earlier.

The competing products all have an many examples of real life autotrading examples and eSignal has none.

I thought it would be in all our interests to take this forum to the next level and without giving away any "secrets" posting a "dummy" strategy (or viable one as more heads are better then one.) would be beneficial.

I have one but need to clean up the code as I'm not a developer and my code is nowhere as tight as yours in terms of validating connections, etc.


glen