multi-Interval reference

[tilmon]

My question is how would I reference a time interval other than the one I am in on a script. Let's say I am in "interval() =2 minute chart", but I want to compare an MAStudy in this interval with one in a 5 minute interval, i.e. "interval() = 5 minutes". I know how to do all the linear extrapolations and all to create a slower moving MA. But, that is not the issue. In general, I am wondering how to refer from within a given time interval to "outside" that interval in one script, in order to compare things from different time intervals wthin one script (which are not always easy to do linear extrapolations on, like Boolean conditions etc.. Would I use an efs.external Call (a separate function with a different interval at the end of the script, below "main" or what. I've never done it, and am new enough I haven't got a clue if it can even be done, or is forbidden in EFS. Does everything in a script used on one chart have to conform to a single time interval?
Are there any examples you can refer me to. I am sure I could pick it up if I have someplace to start, if its is permitted.

thx,
Tilmon

[eSignal_JasonK]

Re: multi-Interval reference

Hello Tilmon,

Originally posted by tilmon
My question is how would I reference a time interval other than the one I am in on a script. Let's say I am in "interval() =2 minute chart", but I want to compare an MAStudy in this interval with one in a 5 minute interval, i.e. "interval() = 5 minutes". I know how to do all the linear extrapolations and all to create a slower moving MA. But, that is not the issue. In general, I am wondering how to refer from within a given time interval to "outside" that interval in one script, in order to compare things from different time intervals wthin one script (which are not always easy to do linear extrapolations on, like Boolean conditions etc..

To access external intervals you use the inv() function as a parameter when initializing a series such as the sma( length [, source | sym() | inv()] [, barIndex] ). For any of the built-in studies, when you see a parameter that has the option of using inv(), is where you specify the interval that you want the series to be based on. Here's a basic example. Try this on a 1 or 5 minute chart.

function main() {
    return sma(10, inv("10"));
} 

If you want to write conditions against this MA you can initialize the series like below.

var xMA = null;

function main() {
    if (xMA == null) xMA = sma(10, inv("10"));

    var myMA = xMA.getValue(0);

    // write conditions based on the external MA
    if (myMA ......) {

    }

    return getSeries(xMA);
} 

However, in the above example, keep in mind that the condition will be evaluating the external MA every tick in real time, so any intra-bar or intra-interval signals that that are triggered may not be generated by the same interval later on if the formula is reloaded because historical processing only can evaluate the completed bar information. If you want to write back testing formulas or generate entry signals based on the external interval I recommend that you use getBarStateInterval() and only evaluate for these types of signals (or conditions) upon the closing of the higher time-frame interval. Here's a basic example, which looks for BARSTATE_NEWBAR of the higher interval, which is the first trade of the new bar, and then evaluates the condition based on bar -1. Bar -1 at this bar state is the bar that just closed, which allows the conditions to be based on the close of that bar. This logic for entry signals will allow you to back test these entries as well as generate the same signals in real time.

var xMA = null;

function main() {
    if (xMA == null) xMA = sma(10, inv("10"));

    var myMA = xMA.getValue(-1);

    // write conditions based on the external MA
    if (getBarStateInterval("10") == BARSTATE_NEWBAR && myMA ...your condition...) {

    }

    return getSeries(xMA);
} 

Of course you can also use this on a 2 minute chart to access a 5 minute interval, but I recommend using intervals that are evenly divisible during your development process so that you can get a better visual understanding of the plotted results for the higher-time frame series.

For more complete code examples that utilize inv(), check out the formulas that installed with eSignal in the \Formulas\EFS 2 Custom\ folder. I would also recommend that you perform some searches here in the forums on the "inv" keyword and others that you are interested in learning more about. In addition to the documentation in the EFS Knowledgebase there are many discussions that occur here in the forums that cover these items that you may find helpful and educational. Many questions that you have may be answered by other threads.

Would I use an efs.external Call (a separate function with a different interval at the end of the script, below "main" or what. I've never done it, and am new enough I haven't got a clue if it can even be done, or is forbidden in EFS. Does everything in a script used on one chart have to conform to a single time interval?
Are there any examples you can refer me to. I am sure I could pick it up if I have someplace to start, if its is permitted.

thx,
Tilmon

The efsExternal() function is used to create a series based on the returned data of another formula. In can be used from within main() or any user-defined function that you create. Within a single formula, you can have varies series all based on different intervals up to seven different intervals. By passing inv() as the last parameter you can designate the resulting series of an external formulas calculation on your desired interval. Check out the examples by Alexis in this thread.

[demarcog]

Re: Re: multi-Interval reference

Originally posted by JasonK
[B]Hello Tilmon,



To access external intervals you use the inv() function as a parameter when initializing a series such as the sma( length [, source | sym() | inv()] [, barIndex] ). For any of the built-in studies, when you see a parameter that has the option of using inv(), is where you specify the interval that you want the series to be based on. Here's a basic example. Try this on a 1 or 5 minute chart.

function main() {
    return sma(10, inv("10"));
} 

If you want to write conditions against this MA you can initialize the series like below.

var xMA = null;

function main() {
    if (xMA == null) xMA = sma(10, inv("10"));

    var myMA = xMA.getValue(0);

    // write conditions based on the external MA
    if (myMA ......) {

    }

    return getSeries(xMA);
} 

However, in the above example, keep in mind that the condition will be evaluating the external MA every tick in real time, so any intra-bar or intra-interval signals that that are triggered may not be generated by the same interval later on if the formula is reloaded because historical processing only can evaluate the completed bar information. If you want to write back testing formulas or generate entry signals based on the external interval I recommend that you use getBarStateInterval() and only evaluate for these types of signals (or conditions) upon the closing of the higher time-frame interval. Here's a basic example, which looks for BARSTATE_NEWBAR of the higher interval, which is the first trade of the new bar, and then evaluates the condition based on bar -1. Bar -1 at this bar state is the bar that just closed, which allows the conditions to be based on the close of that bar. This logic for entry signals will allow you to back test these entries as well as generate the same signals in real time.

var xMA = null;

function main() {
    if (xMA == null) xMA = sma(10, inv("10"));

    var myMA = xMA.getValue(-1);

    // write conditions based on the external MA
    if (getBarStateInterval("10") == BARSTATE_NEWBAR && myMA ...your condition...) {

    }

    return getSeries(xMA);
} 

Jason,

I'm running into a situation where using a multiple interval strategy my backtested results are dramatically different (and not for the better) from those derived in realtime. To oversimplify lets say trading signals are based on an indicator or price value increasing or decreasing from the prior value, ie., high(0) > (high(-1), or a simple indicator say an basic oscillator, ie., osc.getValue(0) > osc.getValue(-1)

Are you saying that even if you:

1. use the getBarStateInterval() of the highest time interval

2. evaluate entry based on barindex( 0) barindex( -1) indicator comparasion

3. place orders in the backtester using Strategy--Market-NEXTBAR.....

That the results achieved using the backtester will still not be comparable to those achieved in realtime and the "forward looking" results are not completely eliminated?



And the reason is because the values evaluated by the barindex (0) using historical bars evaluate to a static value (and usually in your favor if a trade was triggered by the bar) yet in realtime will fluxuate with each tick and cause inconsistencies (usually inflated)between backtested and realtime results?

I thought that using EITHER 0 and -1 and strategy-nextbar, OR -1 and -2 using strategy-thisbar were essentially the same for analyzing a strategy?

If I want to my strategies to be as easily transferable from backtesting on historical, bar replay, tick replay, and most importantly in realtime also using multiple time intervals, then is the recomendation to evaluate on -1 and -2 and use strategy-market-thisbar for the backtester and simply buyMarket() (using the generic broker functions.

What I'm slightly confused about is that on historical bars you never really have a barindex(0) condition as that is the test we do to detrmine if we are running realtime ie.,

 if (getBarstate() == BARSTATE_NEWBAR && getBarindex() == 0) bRealtime = true; 

Yet doing a test in the backtester for:

 if osc.getvalue(0) > osc.getvalue(-1) strategy.doLong("buy", Strategy.Market, Strategy.Nextbar) 

Is an acceptable way of comparing values and the osc.getvalue(0) call yields the value of the last "completed" historical bar used by the backtester and/or the getBarState() == BARSTATE_NEWBAR raised condition.

You mentioned that in realtime for multiple time intervals when getBarStateInterval() is raised bar -1 is the last bar just completed....so should different techniques for comparison of indicators be used for historical versus realtime processing?

I realize that becasuse something prints or trades at a given level during a tick replay or backtest, doesn't necessarily mean that we would have executed a trade at that level (plus commission and slippage) were our system trading realtime.

But I would like as closely as possible like to program my strategies in such a way (even if it means having different versions for backtesting and realtime) that the backtested results will be a close as possible to results achieved in realtime and appreciate your giudance on doing so?

If you run across any sample realtime especially multiinterval strategies please let us know as the examples you provided on the backtester have been a tremendous help to me and many others.

There are alot of issues that you and Alexis have probably encountered in the past and myself and many others will be encountering as we expand our collective efforts and EFS strategy development into realtime, multiple interval, automated trading, etc and I for one would be grateful to see samples incorporating these newer elements of EFS.

As always all you help is greatly appreciated..

glen

glen

[eSignal_JasonK]

Re: Re: Re: multi-Interval reference

Hello demarcog,

Originally posted by demarcog
Jason,

I'm running into a situation where using a multiple interval strategy my backtested results are dramatically different (and not for the better) from those derived in realtime.

Remember that if your real time system is evaluating the trade signals on a tick-by-tick basis you will not be able to exactly duplicate the same trade results on a back testing basis. Back testing only processes completed bars. All of the intrabar values of the indicators you may be using in the real time system cannot be accessed during back testing on the historical bars. Only the last, or closing, value for each bar on a historical basis is available for evaluating when back testing. If you want a real time system's trades to match back testing results, then the real time conditions can only be evaluated on completed bars. In real time, when the higher time-frame interval has completed, getBarStateInterval(20) will return BARSTATE_NEWBAR. At that instance all of your conditions need to look at bar -1 and/or older. You cannot look at bar 0 in this case as that will correspond to the opening values of the new 20-min bar (which is not a completed bar) or whatever higher time frame you are looking at. When back testing, looking at bar 0 corresponds to the closing values of the bar, because we are processing completed bars.

To oversimplify lets say trading signals are based on an indicator or price value increasing or decreasing from the prior value, ie., high(0) > (high(-1), or a simple indicator say an basic oscillator, ie., osc.getValue(0) > osc.getValue(-1)

Again, the problem here is that you are trying to evaluate bar 0 and bar -1 for this condition. For the real time system to match back test results, you need to evaluate bar -1 and bar -2 at the newbar state of the higher interval.

Are you saying that even if you:

1. use the getBarStateInterval() of the highest time interval

2. evaluate entry based on barindex( 0) barindex( -1) indicator comparasion

3. place orders in the backtester using Strategy--Market-NEXTBAR.....

That the results achieved using the backtester will still not be comparable to those achieved in realtime and the "forward looking" results are not completely eliminated?

That's correct, which is because of the reference to bar 0 as explained above.

And the reason is because the values evaluated by the barindex (0) using historical bars evaluate to a static value (and usually in your favor if a trade was triggered by the bar) yet in realtime will fluxuate with each tick and cause inconsistencies (usually inflated)between backtested and realtime results?

Again, you're understanding is correct, but try to remember that the bar 0 reference in real time at the newbar state corresponds to the opening values of a bar whereas in back testing it will correspond to the closing values of a bar at the newbar state. It's not really an inconsistency but simply a difference is logic required to process historical bar vs. real time data on a tick-by-tick basis.

I thought that using EITHER 0 and -1 and strategy-nextbar, OR -1 and -2 using strategy-thisbar were essentially the same for analyzing a strategy?

This is true, but for for back testing on completed bars only.

If I want to my strategies to be as easily transferable from backtesting on historical, bar replay, tick replay, and most importantly in realtime also using multiple time intervals, then is the recomendation to evaluate on -1 and -2 and use strategy-market-thisbar for the backtester and simply buyMarket() (using the generic broker functions.

Yes, this is the correct logic.

What I'm slightly confused about is that on historical bars you never really have a barindex(0) condition as that is the test we do to detrmine if we are running realtime ie.,

On a historical basis, getCurrentBarIndex() will not return 0 until you've reached the current real time bar. Any function, such as high(0), where a bar index of 0 is specified refers to the bar index that is relative to the current bar index that is being processed. For example, if the formula is loading and it processes bar -100, high(0) refers to that bar's high. high(-1) would refer to the high of the prior bar on bar -101. When the formula has completed loading and is processing the real time bar, or bar 0, high(0) refers to that bar's high and high(-1) refers to the prior bar's high on bar -1. Hope this help clear up the confusion.

 if (getBarstate() == BARSTATE_NEWBAR && getBarindex() == 0) bRealtime = true; 

Yet doing a test in the backtester for:

 if osc.getvalue(0) > osc.getvalue(-1) strategy.doLong("buy", Strategy.Market, Strategy.Nextbar) 

Is an acceptable way of comparing values and the osc.getvalue(0) call yields the value of the last "completed" historical bar used by the backtester and/or the getBarState() == BARSTATE_NEWBAR raised condition.

You mentioned that in realtime for multiple time intervals when getBarStateInterval() is raised bar -1 is the last bar just completed....so should different techniques for comparison of indicators be used for historical versus realtime processing?

Formula logic required for processing only on a historical basis can be simplified compared to what the real time equivalent would be in most cases. So in that sense, yes, you could say your using different sets of formula logic. However, they can accomplish the same results if both sets of logic are always referencing completed bars (i.e. bar -1 and older). If a system is allowed to evaluate bar 0 on a trade-by-trade basis in real time you will most likely be dealing with false signals. I think this is the central problem you are having. While a bar is building in real time and is trading what will end up being the high for the interval, a particular trade signal may be triggered, or evaluate to true. However, by the time the interval closes, the last price could be at or near the low for the interval. At that price level the trade condition is false. If you allow the trade signal to be taken intrabar (i.e. before the bar has completed) near the high, you are taking an unconfirmed trade signal. When this same condition is evaluated on a historical basis for back testing and only the closing values of the bar are evaluated, no trade signal will occur on that bar. Thus creating a difference between historical back testing results, which are evaluated on confirmed data (completed bars), and real time results that are being allowed to entered on false signals (or unconfirmed data). It's not necessarily wrong to take a trade based on unconfirmed data (false signals), but you need to be aware that those results will not be able to be back tested because the intrabar values of the indicators are not available on a historical basis.

If your oscillator series in the above example is based on a higher time frame, say 10 minutes, and this condition is being evaluated on a 1-minute chart, referencing bar 0 is going to create false results because it is forward looking logic. If the formula is processing the first 1-minute bar of the higher time frames 10-minute window bar 0 of the higher time frame oscillator refers to the closing value of the higher time frame interval which is 9 minutes into the future compared to the current 0 bar reference for the 1-minute bar that is being processed. To avoid this problem always look at bar -1 or older of the higher time frame bar. You should also look at bar -1 for the 1-minute data as well. Always look for the newbar state of the highest interval and then evaluate completed bar data and you'll be fine. If you continue to try to reference bar 0 for back tests and/or real time systems you will continue to run into this source of your confusion.

I realize that becasuse something prints or trades at a given level during a tick replay or backtest, doesn't necessarily mean that we would have executed a trade at that level (plus commission and slippage) were our system trading realtime.

But I would like as closely as possible like to program my strategies in such a way (even if it means having different versions for backtesting and realtime) that the backtested results will be a close as possible to results achieved in realtime and appreciate your giudance on doing so?

As long as both your back testing formula and real time formula are evaluating completed bar data, they should match. However, trying to get results to match for a real time system that takes false signals compared to the same conditions on a back testing basis is an unrealistic expectation to have. In order for this scenario to match up, the back testing formula would have to be able to drill down and evaluate each trade that occurred on each historical bar, which does not happen. Even if we could make this happen it would limit the back test to 10 days of data because that is the longest period available for tick data.

If you run across any sample realtime especially multiinterval strategies please let us know as the examples you provided on the backtester have been a tremendous help to me and many others.

There are alot of issues that you and Alexis have probably encountered in the past and myself and many others will be encountering as we expand our collective efforts and EFS strategy development into realtime, multiple interval, automated trading, etc and I for one would be grateful to see samples incorporating these newer elements of EFS.

As always all you help is greatly appreciated..

glen

glen

Again, thank you for your kinds words and suggestions. Hope you find this information helpful.

[demarcog]

Jason,

As always I can't thank you enough for such a comprehensive, thoughtful and extremely interesting response.

I found it extremely helpful and will find myself reading it many times, learning something new each time.

It raises some interesting design questions that I need to think about in terms of:

If I'm looking for a certain synchronization between a 5 minute, 20 minute and 60 minute interval but only do so on getBarStateInterval("60") (which is basically an end of hour/once per hour situation ).......

Am I actually "missing" potential trades when all three were "lined up correctly" but because it wasn't on the 60 minute boundary completion had to wait for for the next occurance?

Or am I misunderstanding the significance of "waiting" the 60 minutes?

Isn't the technical basis or "theory" behind most traders use of multiple time frames: got a signal, on the 5 minute, check the 20 minute, then check the 60 minute, for each occurance of a signal generated on a 5 minute chart. Having confirmation on higher, "more powerful" larger time frames certainly can't hurt a systems overall performance?

Or alternatively, is it the longer term that is checked first, ie. the 60 minute and don't check it again until the next 60 minute bar's completion, then check the 20 minute (every 20 mins), then the 5 minute? This sounds more like what the getBarInterval("60") is accomplishiing.


Or am I misunderstanding the significance of "waiting" for the 60 minute bar completion?

Would it make sense to programatically somehow force or simulate a "BARSTATE_NEWBAR" condition on the higher 60 minute bars data every 5 and/or 20 minutes but contain 60 minutes "worth"of data?

Anyway just some questions prompted by your awesome reponse...I guess I'm thinking out loud and would appreciate hearing from anyone who may have developed systems on multiple time frames ..

From the little research I've done and obviolusly my results are questionable at this point, but in case someone is interested, it looks like there may indeed be some validity to the notion that a system applied to multiple time frames does perform better then the same system applied independently to those same time frames.

thanks again


glen

[eSignal_JasonK]

Hello demarcog,

Originally posted by demarcog
Jason,

As always I can't thank you enough for such a comprehensive, thoughtful and extremely interesting response.

I found it extremely helpful and will find myself reading it many times, learning something new each time.

It raises some interesting design questions that I need to think about in terms of:

If I'm looking for a certain synchronization between a 5 minute, 20 minute and 60 minute interval but only do so on getBarStateInterval("60") (which is basically an end of hour/once per hour situation ).......

Am I actually "missing" potential trades when all three were "lined up correctly" but because it wasn't on the 60 minute boundary completion had to wait for for the next occurance?

Or am I misunderstanding the significance of "waiting" the 60 minutes?

Your system probably wouldn't miss any trades, but it would take some additional trades that were based on the unconfirmed (false) intrabar signals.

Again, as I've stated previously, it's not necessarily wrong to take a trade based on unconfirmed data (false signals), but you need to be aware that those results will not be able to be back tested because the intrabar values of the indicators are not available on a historical basis. You can certainly trade your real time system using the intrabar signals if that is your preference. I just want you to understand why you will not be able to back test a system that is based on these intrabar signals.

Isn't the technical basis or "theory" behind most traders use of multiple time frames: got a signal, on the 5 minute, check the 20 minute, then check the 60 minute, for each occurance of a signal generated on a 5 minute chart. Having confirmation on higher, "more powerful" larger time frames certainly can't hurt a systems overall performance?

Or alternatively, is it the longer term that is checked first, ie. the 60 minute and don't check it again until the next 60 minute bar's completion, then check the 20 minute (every 20 mins), then the 5 minute? This sounds more like what the getBarInterval("60") is accomplishiing.

This is more of a matter of preference of the individual trader. In terms of back testing and real time formulas producing matching results in EFS, the latter is the way to go.

Or am I misunderstanding the significance of "waiting" for the 60 minute bar completion?

Would it make sense to programatically somehow force or simulate a "BARSTATE_NEWBAR" condition on the higher 60 minute bars data every 5 and/or 20 minutes but contain 60 minutes "worth"of data?

This is not possible because you can't cheat time. By definition, a 60 minute interval is only going to have a new bar once every 60 minutes. If you want to base trades on a 60-minute interval, then trade the 60-minute interval. If you want to evaluate trades more frequently, then trade lower intervals.