Bactest Results versus Real World

[willr]

I'm new to strategy backtesting, so don't shoot me if this is a stupid post!

I sent this email to support about what appears to be a limitation with the backtest report -- any of you gurus have some simple solutions for me?

-----original message---------
I have a simple issue with every strategy I program and backtest with esignal.

Problem: The entry/exit signals that are being generated "intra-bar" during real-time trading with esignal do NOT show up on the backtest report.

These "false/early signals" would actually be the "real" entry/exit points when trading live because the calculations are running tick by tick (as it should). Esignal properly plots and triggers these alerts real-time.

The backtest report however, is not calculating this tick data; it
instead appears to use the OHLC of each bar and plots the entry/exit signals as such only after the bar closed.
This gives me a useless report that is over-optimized because all of the "false" signals have been removed from the report.

Example: Say we use a simple MACD crossover as our entry/exit signal.
This signal can be generated on an interval bar (3,4,5,6min etc)
several times during the same bar until that bar closes. The backtest report however would list this as only a single entry/exit signal entry.

More importantly using the same example, a MACD crossover can occur intra-bar (generating a signal real-time), but by the time the bar closes the price movement intra-bar has undone the MACD crossover. The backtest report looking historically does not even reflect this signal ever occurred!

Without knowing what the "real" entry/exit points were, I can't
generate an accurate backtest report to see how much $ could
potentially be made/lost.

Workaround: I have to change the strategy exit from THISBAR to NEXTBAR to more closely match the backtest results with the real world. This of course strips out most of the profit from the system and is just plain inefficient. This is not a solution.

Solution: I need to generate an accurate strategy report that uses "THISBAR" as the entry/exit points. There should be an option to run the report against actual tick data. This would return a result set displaying all of the signals intrabar.

This is why I am subscribing to esignal, and I've reached a stopping point in my research because of this apparent limitation.
I observe this reporting issue with MACD crossover, Stochastic band crossover, RSI band crossover etc.
How can anyone generate an accurate report on the above signals with only closed bar calculations?
How can I accomplish this with the esignal software?

I must be doing something wrong -- please help!
Thank you for your time.
-----original message---------

Thanks in advance all.
-Willr

[ACM]

Will

While I can see your point (albeit not agreeing with it) I also see that you make some broad statements that are not universally applicable.

These "false/early signals" would actually be the "real" entry/exit points when trading live because the calculations are running tick by tick (as it should)

Not necessarily. Some use setComputeOnClose() in their systems thereby triggering orders ONLY once a bar is closed. In that case the trading strategy is correctly reflected in the Back Tester and does indeed represent the real world.
Furthermore some use entries on stops generated for example on the high or the low of the setup bar being taken out. Again these entries are properly and correctly reflected in the Back Tester. No false or missed signals there.
Lastly others use limit orders again preventing false signals and once more the Back Tester will correctly reflect the real world trading environment.

More importantly using the same example, a MACD crossover can occur intra-bar (generating a signal real-time), but by the time the bar closes the price movement intra-bar has undone the MACD crossover. The backtest report looking historically does not even reflect this signal ever occurred!

This is exactly the reason why some will use only orders generated on the close of a bar rather than intrabar so as to avoid those false early signals.

Workaround: I have to change the strategy exit from THISBAR to NEXTBAR to more closely match the backtest results with the real world. This of course strips out most of the profit from the system and is just plain inefficient. This is not a solution

I think you will find many will strongly argue this statement and will instead sustain that a signal on close or a stop or limit strategy is far more profitable as it avoids false entries.
Some will go even further by suggesting that a system/strategy that is only successful when a bar is being built is not one that will do well over time. True that on some systems you can perhaps eek out some extra profit by doing this, but they also stand alone when trading on the close of bar.
I also find interesting the view that not being able to pick up early signals results in missed profits and not necessarily in extra losses.

How can anyone generate an accurate report on the above signals with only closed bar calculations?

Again I believe you are making a broad statement. The fact that this may not fit your needs is one thing, but to then generalize is another as what you are saying is not applicable to all traders. AFAIK there are plenty succesful systems out there that do not rely at all on intrabar values on the contrary they make a point of generating orders outside of the set up bar. Most of the volatility based systems I know tend to do that and with those systems the price is not only known well in advance but can easily be entered in the system preventing any kind of false signals.

It goes without saying that all the above is just another opinion.

Alex

[willr]

Hey thanks for the post Alexis!

All broad generalizations aside, my inquiry is not as much to debate strategy but to understand if there is a way to make the "false/early" intra-bar signals show up on the backtest report -- a setting of some sort that maybe I'm missing??
I think your confirming that "No", there is not a way to do this so to design the strategy in a different way such as use "setComputeOnClose()" or stop/limit or other closebar entry methods.

I'm fine with this.

However, my intent of seeing this data in the backtest report was then to refine/improve the strategy and attempt reduce these early/false alarms in a creative fashion by implementing/testing other threshold criteria. But if it does not show up in the report, and I'm convinced now there is not a way to make this show up, then I guess I'll try some of the other ideas you've suggested. I'm surprised I'm the only one to complain about such an issue -- but I told you I was new to backtesting!

Happy trading!
-Willr

[gspiker]

Will,

The biggest problem is that the data vendors would have to keep tick level data for all the historical data they have. So imagine going back far enough for a daily system to be able to do serious backtesting, then multiply that by the number of symbols you support - it is a huge amount of data.

Sometimes packages try to get around this by buiding in simulators, but I'm not convinced that backtesting on a simulator doesn't just allow you to veryify that your code works really well with whatever model the vendors used to simulate the market.

In any case, while it would be nice to have, I agree with Alex it isn't all that important. A good system will backtest well on a closing basis, and if you want earlier/quicker entries just run the same system on a lower interval chart...it will signal you in and out more quickly.

[ACM]

Will
If you wanted to approximate, on an average basis, an intrabar entry there could be a couple of solutions you might want to try.
The first one would be to use either the midpoint of a bar or the the midpoint plus, for example, a 2 period moving average of the midpoint so as to provide some bias to the upside or downside depending on market direction. This would then be the price the Back Tester uses for entries/exits.
The other one would be to compute where price needs to be on the next bar for a crossover of two averages to happen (this would apply to your MACD example). This way you could set that price as the entry/exit if the close touches it.
Again both methods are approximations and do not resolve the issue of false signals. Also it is very possible that neither solution can be made to fit in the logic of your system or is what you need.
Just an idea
Alex

[willr]

Good information folks ... thanks for the posts!
-Willr

[SmoothOperator]

I have got around this problem in my strategies with slippage adjustment. I use some figures wich correlate with average bar range as a crude approximation for slippage.

I agree that you don't necessarily need tick-by-tick data for establishing the successfulness of any given formula, but wouldn't it be nice to have it? Backtesting would return much more realistic results, I have to agree with Will.

As far as "it is too much data to handle for data provider" -- please! Were are not living in 80's anymore! If quote.com can provide some 2 years worth of intraday data (which includes options btw) then 60 or 90 days worth of tick-by-tick should not be a problem for eSignal, either. It might take awhile to download such data, but still...

Just my opinion.