How to incorporate completed bars and current bars price action into strategy?

[demarcog]

Is there a recommended way or example available on how to structure a strategy efs that evaluates trade entry conditions based on indicator values for the prior bar (-1) but also evlauates price action for the current bar for entry and exit conditions that will work for back testing, realtime and tick replay?

I usually encapsulate my strategy in a _NEWBAR loop and set a flag for bRealtime to query realtime or tick replay to execute generic broker calls or the strategy methods.

I also always use -1 on completed bars (setup bar) for indicator entry but need to be able to process trade by trade in realtime and tick replay for the actual trade entry trigger and also for management of stops and profit targets.

Not sure if I should execute the entire efs outside of _NEWBAR control which I thought would work and seems fine in a back test, but is inefficient?

I'm also seeing strange results in tick replay with or without SetComputeOnClose, which I thought would only pass control to the efs upon bar completion?

I normally evaluate indicators on -1 but also added a "trend test" of close(0) above a EMA and knew when running realtime that false signals could occur intrabar.

Does processing tick by tick have a seemingly disporportionate effect on indicator values?

Normally when using a 15 minute chart interval that would have "included" say 100 ticks, and visually if watching the 15 minute chart, CCI and EMA values "seem" to change gradually over time in an incremental fashion.

Intuitively I thought a 1 tick trade, which could represent 1 percent of the prior bars activity has a disporpotional effect on the values returned?


Thanks,

Glen

[eSignal_JasonK]

Re: How to incorporate completed bars and current bars price action into strategy?

Hello Glen,

Originally posted by demarcog
Is there a recommended way or example available on how to structure a strategy efs that evaluates trade entry conditions based on indicator values for the prior bar (-1) but also evlauates price action for the current bar for entry and exit conditions that will work for back testing, realtime and tick replay?

I usually encapsulate my strategy in a _NEWBAR loop and set a flag for bRealtime to query realtime or tick replay to execute generic broker calls or the strategy methods.

I also always use -1 on completed bars (setup bar) for indicator entry but need to be able to process trade by trade in realtime and tick replay for the actual trade entry trigger and also for management of stops and profit targets.

Please see my reply to you in this thread, which seems to be the same topic.

Not sure if I should execute the entire efs outside of _NEWBAR control which I thought would work and seems fine in a back test, but is inefficient?

This works in back testing because the EFS only executes once per bar, which is the _NEWBAR state. In back testing, you don't have to check for _NEWBAR explicitly because the formula is only executed once per bar. The real time formula code needs to look for the _NEWBAR state to simulate the same execution logic that occurs in back testing.

I'm also seeing strange results in tick replay with or without SetComputeOnClose, which I thought would only pass control to the efs upon bar completion?

That is what it does. If you are adding and removing setComputeOnClose() from your formula code, you may also need to remove and then reapply the formula to the chart in order for the formula to be initialized properly. However, my recommendation is to not use setComputeOnClose(). Use bar states to control your formula code execution for real time or tick-by-tick processing.

I normally evaluate indicators on -1 but also added a "trend test" of close(0) above a EMA and knew when running realtime that false signals could occur intrabar.

Does processing tick by tick have a seemingly disporportionate effect on indicator values?

I'm not sure what you mean by disproportionate effect. Can you explain in greater detail? Perhaps include some chart images to illustrate?

Normally when using a 15 minute chart interval that would have "included" say 100 ticks, and visually if watching the 15 minute chart, CCI and EMA values "seem" to change gradually over time in an incremental fashion.

Intuitively I thought a 1 tick trade, which could represent 1 percent of the prior bars activity has a disporpotional effect on the values returned?


Thanks,

Glen

If the source for the CCI and EMA is using the close series (which is their default), then the value of the indicators on the real time bar will update accordingly as the close price changes. Again I'm not sure what you mean by disproportional effect.

Try this exercise, which may help sort this out for you. Using the built-in studies from the Basic Studies menu, apply a CCI and an EMA to your chart and set the source to the high series. Then watch what happens in real time to the values of the indicators. You will only see them update when a new high occurs. Let me know if this helps.

[demarcog]

Re: Re: How to incorporate completed bars and current bars price action into strategy?

Originally posted by JasonK
Hello Glen,



Please see my reply to you in this thread, which seems to be the same topic.

Hi Jason,

Thanks for the response and your earlier invaluable explanations regarding the discrepencies between back testing and realtime. My lack of sleep trying to reslove this seems to have contributed to the lack of clarity in my post for which I apologize.

The discrepencies I was seeing which I should have explained went way beyond the minimal discrepencies you described. The number of trades, P&L etc. were way beyond the expectable differences you described.

I'm attaching a sample strategy to illustrate my question. This SAR strategy currently is based on the close of the previous bar and is back tester and realtime compatable.

What changes would have to be made to alter this strategy to not only run realtime, but to initiate a position on a 1 tick penetration of the indicated SAR value?

Altering the Strategy statements to use STOP THISBAR using the SAR + or - 1 tick is straight forward and will closely approximate in a back test the realtime results, of course accounting for differences you described.

I was unsure about how to structure a strategy in terms of when indicators are evaluated based on completed -1 bars, but looking for a realtime trade 1 tick above below a level:

- does some portion of code that looks for a 1 tick penetration above/below a certain price variable have to be contained in a seperate and explicit BARSTATE_CURRENTBAR loop structure?

- does the logic of the entire strategy have to be duplicated under a BARSTATE_NEWBAR controlled block of code for backtesting and also under BARSTATE_CURRENTBAR controlled block of code?

- or is there some other method of removing _NEWBAR && _CURRENTBAR loop control enabling the entire efs to run on a trade by trade basis?

To use an example of the attached SAR strategy, I know the SAR value based on the -1 evaluation of the SAR indicator. If I wanted to have a position initiated on a 1 tick penetration on a realtime basis, what changes would have to be made.

This works in back testing because the EFS only executes once per bar, which is the _NEWBAR state. In back testing, you don't have to check for _NEWBAR explicitly because the formula is only executed once per bar. The real time formula code needs to look for the _NEWBAR state to simulate the same execution logic that occurs in back testing.

So you are saying that without the _NEWBAR test in realtime or tick replay the results will not be similiar between a back test and tick replay or realtime execution? That alone could be the reason I was seeing such drastically different results.

That is what it does. If you are adding and removing setComputeOnClose() from your formula code, you may also need to remove and then reapply the formula to the chart in order for the formula to be initialized properly. However, my recommendation is to not use setComputeOnClose(). Use bar states to control your formula code execution for real time or tick-by-tick processing.

I wasn't aware of that, thanks for the heads up. took your suggestion and elimnated SetComputeOnClose().

Proper timing of when a formula is reloaded during a tick replay is also something I encountered, as global variables such as vPosition are reset and obviously getting out of sync will cause erroneous results.

I'm not sure what you mean by disproportionate effect. Can you explain in greater detail? Perhaps include some chart images to illustrate?

I'll try and capture some images. If you take a daily bar chart for example, that would encompass perhaps thousands of ticks. Then when running realtime to have a few ticks cause the indicator values to chage so dramatically, cauisng so many "false signals" is just something that suprised me as intuitively I thought that some "weighting" for the fact that an indicator reading on a daily chart representing thousands of ticks has the same "weight" as a single tick on a realtime chart.

If the source for the CCI and EMA is using the close series (which is their default), then the value of the indicators on the real time bar will update accordingly as the close price changes. Again I'm not sure what you mean by disproportional effect.

Try this exercise, which may help sort this out for you. Using the built-in studies from the Basic Studies menu, apply a CCI and an EMA to your chart and set the source to the high series. Then watch what happens in real time to the values of the indicators. You will only see them update when a new high occurs. Let me know if this helps.

I do see what you mean, I was using hlc3 for my price source which looks even more volatile.



Thanks very much for all your assistance I really appreaciate it very much.

Glen

[eSignal_JasonK]

Re: Re: Re: How to incorporate completed bars and current bars price action into strategy?

Hello Glen,

Originally posted by demarcog
Hi Jason,

Thanks for the response and your earlier invaluable explanations regarding the discrepencies between back testing and realtime. My lack of sleep trying to reslove this seems to have contributed to the lack of clarity in my post for which I apologize.

You're most welcome as always.

First, let me make a couple suggestions.

1) Your Enable parameter is set to true by default, which could allow a real time trade to submit at the completion of the formula loading process. This could be undesirable. I recommend that you set that parameter to false by default and allow the eSignal and your formulas to complete the initialization process. Once everything is running and your ready to test, then set that parameter to true.

2) I would also recommend using two separate formulas. One for back testing and one for real time analysis. Because of the two different sets of code logic needed, trying to combine them into a single formula adds a layer of coding complexity. This is an extra layer that is not necessary in my opinion, which only adds to the complexity of the code and development time.

The discrepencies I was seeing which I should have explained went way beyond the minimal discrepencies you described. The number of trades, P&L etc. were way beyond the expectable differences you described.

The difference in the number of trades would be directly related to the number of historical bars analyzed in the back test vs the number of bars tested in real time. If these two sets of bars are not the exact same set of bars, the number of trades and P&L should be expected to be different.

I'm attaching a sample strategy to illustrate my question. This SAR strategy currently is based on the close of the previous bar and is back tester and realtime compatable.

What changes would have to be made to alter this strategy to not only run realtime, but to initiate a position on a 1 tick penetration of the indicated SAR value?

Altering the Strategy statements to use STOP THISBAR using the SAR + or - 1 tick is straight forward and will closely approximate in a back test the realtime results, of course accounting for differences you described.

With this particular strategy, I don't think it is possible. In order to simulate this real time event in back testing, EFS would need to know what the value of the SAR was on the current bar just before it was touched, which causes the SAR to flip to the other side. Because back testing only sees completed bar data, it also only sees the final value of the SAR for any given bar. The next best thing you could do is base your back testing assumptions on the prior bar's SAR value to the current bar's high or low and record the back tested trades with LIMIT/THISBAR constants where the limit price is set to the value of the open or the previous bar's SAR value. Use the open price only if that price breached the previous bar's SAR value.

For the real time logic you can test the current bar's price against the current bar's SAR value because that value is available on a tick by tick basis. This is logic that can only be done in real time.

I was unsure about how to structure a strategy in terms of when indicators are evaluated based on completed -1 bars, but looking for a realtime trade 1 tick above below a level:

- does some portion of code that looks for a 1 tick penetration above/below a certain price variable have to be contained in a seperate and explicit BARSTATE_CURRENTBAR loop structure?

No, in real time, each trade that occurs intra-bar other than the opening trade returns a _CURRENTBAR state with getBarState(). Therefore, it is not necessary to check for this in real time. On a historical bar basis, _CURRENTBAR state never occurs. I don't think you would want to check for this state in real time because it would filter out the opening trades of the real time bars.

- does the logic of the entire strategy have to be duplicated under a BARSTATE_NEWBAR controlled block of code for backtesting and also under BARSTATE_CURRENTBAR controlled block of code?

No. You could simply look at the bar index with getCurrentBarIndex(). If the return value of this function is less than 0, you're processing historical bars. When its 0, you're processing real time.

- or is there some other method of removing _NEWBAR && _CURRENTBAR loop control enabling the entire efs to run on a trade by trade basis?

Historical bars can't be processed on a trade by trade basis for back testing with the Strategy Analyzer. In real time, if you don't restrict a code block from executing based on any of the bar states, the code will execute upon each trade that occurs by default unless you're using setComputeOnClose().

To use an example of the attached SAR strategy, I know the SAR value based on the -1 evaluation of the SAR indicator. If I wanted to have a position initiated on a 1 tick penetration on a realtime basis, what changes would have to be made.

Assuming you want to compare the real time bar price to the previous bar's SAR value then you could compare close(0) to the -1 value of the SAR in your conditions. You may also need to add another flag to the logic to prevent multiple signals from occurring on the real time bar. For an example of this logic see AlertOncePerBar.efs.

Proper timing of when a formula is reloaded during a tick replay is also something I encountered, as global variables such as vPosition are reset and obviously getting out of sync will cause erroneous results.

That should be expected due to the default behavior of EFS and the loading (or reloading) process. If you want to maintain a particular value for a variable that will be unchanged due to a reload, you could use a setGlobalValue()/getGlobalValue() routine that is only allowed to update that value in real time on bar 0.

I'll try and capture some images. If you take a daily bar chart for example, that would encompass perhaps thousands of ticks. Then when running realtime to have a few ticks cause the indicator values to chage so dramatically, cauisng so many "false signals" is just something that suprised me as intuitively I thought that some "weighting" for the fact that an indicator reading on a daily chart representing thousands of ticks has the same "weight" as a single tick on a realtime chart.

Indicator calculations that are using close() or hlc3() as the source do not account for any weighting based on the number of trades that occur within a given bar. With either of these two sources, the CCI or EMA calculations will not care how many trades occurred during the bar. The current value of these studies will be based on the most recent value of close() or hlc3(). Your expectation that these indicator values are weighted by each trade that has occured within the bar interval is incorrect.

[eSignal_JasonK]

Hello Glen,

I've realized I made a mistake on one section of my reply.

With this particular strategy, I don't think it is possible. In order to simulate this real time event in back testing, EFS would need to know what the value of the SAR was on the current bar just before it was touched, which causes the SAR to flip to the other side. Because back testing only sees completed bar data, it also only sees the final value of the SAR for any given bar. The next best thing you could do is base your back testing assumptions on the prior bar's SAR value to the current bar's high or low and record the back tested trades with LIMIT/THISBAR constants where the limit price is set to the value of the open or the previous bar's SAR value. Use the open price only if that price breached the previous bar's SAR value.

For the real time logic you can test the current bar's price against the current bar's SAR value because that value is available on a tick by tick basis. This is logic that can only be done in real time.

This is not completely accurate. The real time value of the SAR will back away from the real time bar up to but not exceeding the previous bar's SAR value. Our SAR study is dynamic on the real time bar, which I had forgotten. Therefore, we can make the assumption in back testing as to what the value of the SAR would have been at the time of the flip. As the flip in real time occurs at the previous bar's SAR value. The back testing condition will test the current bar's high or low against the previous bar's SAR value and record the trade with LIMIT/THISBAR in the same manner I previously mentioned. That part still holds. The real time logic can use the same test, or condition, to trigger the trade as the event occurs. So you could still compare the current bar's high or low to the previous bar's SAR value in real time. The real time logic won't need to specifically test the open price, because at the opening trade of the real time bar, the open, high and low are all the same price. Sorry for the confusion.

[demarcog]

Jason,

Thanks for the response, which are so detailed and helpful that I usually need to read them carefully several times before formulating any questions or issues.

I just wanted to mention that I was looking for a very simple functional/example strategy to use in this post and apparently my choice of the SAR was not the best one due to the complexities you mentioned.

I also didnt realize it initializes the primary SAR value to 0 which distorts the chart scaling and requires a large number of bar to works it's way up into the chart price scale. In addition, there is a bug that the first trade, where there is not a reversal, it still sends out 2 orders, sorry about that.

I put together a very simple and bare bones Oscillator <> 0, strategy that also checks for a price above/below the high/low of the bar where the indicator crosssed 0 which perhaps I could modify and test based on your repsonses.


I initially searched and sure there are some, but didn't recall seeing an example of a Strategy/ Realtime / & tick x tick/ indicator & price action based type of efs.


It would be a tremendous help to me and move my strategy automation forward and perhaps others will benefit by having a workable example available.

Thanks Again,

Glen

 /* Example of basic strategy to illustrate how to code an strategy EFS that is  Back Test, Real Time/Bars, Real Time/Ticks & tick replay capable*/ 

// Global variables
var vPos          =   0;                          // indicates position status, 0, -1, +1                    
var bRealtime     =   false;                      // true if running realtime - flag determines if strategy object or generic broker functions are used
var vOSC1         =   osc(5, 20, false);          // builtin study
var vFixedStop    =   .0050;                      // to use for fixed stop logic....note 4 decimals for currencies
var vFixedProfit  =   .0100;                      // to use for profit target, also 4 decimals
var bBarTrade     =   false;                      // flag to prevent "false signals" when multiple trades occor on same bar
var bAllowRevs    =   false;                      // flag to indicate whether to allow reversals based on indicator 

function preMain() {
  
    setPriceStudy(true);
    setStudyTitle("##s0wizard");

}
/* Strategy - long when OSC value is above/below zero and high/low current realtime price is above/below high/low of prior indicator or "setup bar"
   currently main is called when all bars are loading, when a bar is complete, and tick by tick in realtime.                                       */
function main() {      
 
  if (getBarState() === BARSTATE_NEWBAR && getCurrentBarIndex() === 0) bRealtime = true;

  if (vOSC1.getValue(-2) == null) return;                                    // make sure indicator initializes
  
  if (vOSC1.getValue(-1) > 0 && high(0) > high(-1)) LongCond();
  
    else if ( vOSC1.getValue(-1) <= 0 && low(0) < low(-1)) ShortCond();
  
  if ( vPos === 1)  setBarBgColor(Color.RGB(0,128,128));
  if ( vPos === -1) setBarBgColor(Color.RGB(255,0,0));
  
    return null;
}

function postMain() {
  
}
//* note strategy assumes no price gaps as stop/entry price will always be recorded by strategy object as valid.
function LongCond() {
    if (vPos != 1 && !bRealtime){ Strategy.doLong("", Strategy.STOP, Strategy.THISBAR, Strategy.DEFAULT, high(-1)); debugPrintln("BTL " +high(-1));}
    if (vPos == 0 && bRealtime && getCurrentBarIndex() === 0 ){ buyMarket(getSymbol(), 100); debugPrintln("GBL " +high(-1));}
    if (vPos == -1 && bRealtime && getCurrentBarIndex() === 0 ){ buyMarket(getSymbol(), 100); buyMarket(getSymbol(), 100);debugPrintln("GBL " +high(-1)); }
    //   if (vPos != 1) {setBarBgColor(Color.RGB(0,128,128)); }
    vPos = 1;
}
   
function ShortCond() {
   if (vPos != -1 && !bRealtime) {Strategy.doShort("", Strategy.STOP, Strategy.THISBAR, Strategy.DEFAULT, low(-1));debugPrintln("BTS " +low(-1));}
   if (vPos == 0  && bRealtime && getCurrentBarIndex() === 0 ){ sellShortMarket(getSymbol(), 100);debugPrintln("GBS " +low(-1));}
   if (vPos == +1 && bRealtime && getCurrentBarIndex() === 0 ){ sellMarket(getSymbol(), 100); sellShortMarket(getSymbol(), 100);debugPrintln("GBS " +low(-1)); }
   //   if (vPos != -1){ setBarBgColor(Color.RGB(255,0,0));  }
   vPos = -1;
} 

[demarcog]

Jason,

Thought it might be useful to have a common data file to use when testing the sample EFS strategy.

The attached file is a 3 day tick download of symbol GBP@FXCM A0-FX with and ending data of 07/19/07.

The data was edited to gloablly change each record of the file for all occurances of "Q," to "T," .

FOREX data does as stored in a tick download file does not contain the necessary trade, or "T" records and was edited to enable tick replay to process the data.

Three days of tick data is about 4mb unzipped for GBP@FXCM A0-FX, hopefully it will compress down to the 100k file size attachment limitation.

Glen

[demarcog]

Jason,

After reading your response I spent a day running the simplier strategy I posted in tick replay (with your BarIndex and BarStates example efs's also loaded....which was quite useful to see realtime....and pretty cool also).


The combination of your invaluable and numerous responses to this and earlier posts I made, and the useful BarState/BarIndexes efs's caused the "light" to go on, clearing up any confusion whatsoever I had regarding these issues.

Although not very complex, I had difficulty at first visualizing exactly what was occurring. In hindsight of course, now that I understand, it couldn't be simplier or straight forward and can't imagine anyone not getting it right away!

All the major discrepencies I saw between back test and realtime/tick replay were either my misunderstanding of the JUMP key in Tick Replay or some other "bug" in my EFS.

I can't over estimate how important it was for me to "get this", as the next step for me is autotrading realtime a few basic strategies. Not something that is wise to do without understanding completely what's going on.

I just wanted to thank you very much for this and all the invaluable assistance, time and on this one especially, patience both you and Alex in particular generously provide.

Wheteher these systems trade successfully in the future remains to be seen. What is certain is that without your help I would not have gotten this far, and for that I couldn't be more appreciative.

Thank You,


Glen

[eSignal_JasonK]

Hello Glen,

You're most welcome and thank you very much for your kind words. I'm very happy hear about your progress with your EFS development and wish you the best of luck with your systems.

[zeller4]

DYNAMIC SAR STUDY

JasonK,

Is there a list of the studies that are dynamic (as you discussed with the SAR study)?)

kz

Originally posted by JasonK
Hello Glen,

I've realized I made a mistake on one section of my reply.



This is not completely accurate. The real time value of the SAR will back away from the real time bar up to but not exceeding the previous bar's SAR value. Our SAR study is dynamic on the real time bar, which I had forgotten. Therefore, we can make the assumption in back testing as to what the value of the SAR would have been at the time of the flip. As the flip in real time occurs at the previous bar's SAR value. The back testing condition will test the current bar's high or low against the previous bar's SAR value and record the trade with LIMIT/THISBAR in the same manner I previously mentioned. That part still holds. The real time logic can use the same test, or condition, to trigger the trade as the event occurs. So you could still compare the current bar's high or low to the previous bar's SAR value in real time. The real time logic won't need to specifically test the open price, because at the opening trade of the real time bar, the open, high and low are all the same price. Sorry for the confusion.

[Doji3333]

88

I handle all of this with the same piece of code I've been posting for weeks. I write alot of autotrade and backtest systems for clients and "that one piece of code" can handle all of it.

It's all in how you set it up...

//--------------------------------------------
//  Only runs if it is a new bar
  if (new_bar) {

   // run analysis of last bar for new triggers
   if (close(-1) > open(-1) { 
     BuyMarket...
     graw graphics
     do whatever
   }

  }


//--------------------------------------------
//  Otherwise, run RT stuff here.
//.. this is normally where I run stop checks, pt checks, eod checks, whatever you need to make it work. 

For the RT autotrading stuff, you need to learn to create events and use arrays (which are not always needed). Setting up EVENTS with EFS is not hard, you simply use BOOLEAN (T/F) control variables and set things to go off at certain times or in certain orders.