Back Testing

[gansoro]

Can I back test a system across multiple securities at the same time? How?

Can I back test multiple systems across multiple securities? How?

[gspiker]

Can I back test a system across multiple securities at the same time? How?

Can I back test multiple systems across multiple securities? How?

I think what #1 and #2 what you are aiming at is a way of selecting multiple symbols from a list and saying apply these backtest study's to them. If that is the goal, this isn't possible at this time.

However, going by just what you ask, and not making an assumption about how it is implemented, all you have to do is just to bring up charts of the securities in question and loading the backtest formula on them. One could also imagine tat with a little work you could write a backtest strategy that accepted symbols as input and would test all symbols against the strategy, the problem there is even if care was take to label the long and short by symbol in the study - the report will use all results to generate a single P&L, drawdown, etc...

#2 is also possible in a similar was to #1. Just up multiple chats and load different backtest strategies on them.

[gansoro]

Is it possible, or is it planned in later releases, to be able to back test a strategy/system on a basket of stocks (watch list)... for instance in a quote window? The results for each individual security could be sent to a back testing log file, which could be called up by the trader. This seems like one way to handle the need. What do you think?

[gspiker]

I think it would be great, but by nature of the way the initial implementation of backtesting was done, this would take a rewrite of everything. I don't know if eSignal would want to do that, but there are many advantages for them to make a "backtest lite" that would be simpiler to use and implement features like this.

G

[williamwong]

Backtesting features

I would like to know if the backtesting function in v7.1 has the following features:

a. ability to set the dollar per tick, e.g., to backtest e-mini-nasdaq,
how can I set $20 per tick?

b. can I do pyramiding? for example, could I add more long positions to the current open long positions?

c. how can I access current trade information? For example, I would like to exit after 5 points profit, how could I implement this?

[eSignal_Todd]

Originally posted by gspiker
I think it would be great, but by nature of the way the initial implementation of backtesting was done, this would take a rewrite of everything. I don't know if eSignal would want to do that, but there are many advantages for them to make a "backtest lite" that would be simpiler to use and implement features like this.

G

We're looking into ways to have a second generation backtester that can test a single strategy for a list of symbols or a series of strategies on a single symbol. This would then ideally show a single report that compared the results to help you optimize your strategies.

This is just in the concept phase so might take a while to produce, but we definitely want to continue to increase the power of EFS and the backtesting.

[mattgundersen]

Optimizing

Let's discuss some ideas... I think it might be possible to do optimization on EFS formulas as they work now without a rewrite of the engine.

Remember, we are discussing. There are no committments being made as to when this could or would happen

Let the games begin...

When it comes to optimizing, lets say you are testing something as simple as price cross through a moving average. You might have a 10 bar MA.

But, for optimizing, you want to try 10, 11, 12, 13, 14 bar MA's.

I believe you can do the following in EFS:

var study = null;

function main([b]nLength[/b]) {
    if(nLength == null)
        nLength = 10;

   if(study == null)
       study = new MAStudy(nLength, ....);

   // do backtesting stuff
} 

Now, let's discuss all the WHAT IF's, possibilities, problems, how might we accomplish this... For now, let's also forego discussing the reporting side of things (ie: comparing the results of one system against another).

First, we definitely need to modify the backtester so it can prompt you for optimization parameters. For instance, on the code snippet above, there is the nLength parameter. When the backtester prompts, it would need to ask you the starting value, ending value, and maybe a step/increment.

An example would be: Start: 10, End: 20, Step: 2. This would mean the formula would be run 6 times with nLength(s) of = 10, 12, 14, 16, 18, 20.

If that's the case, we can simply prompt you for number parameters when the EFS is run via the backtester.

The tricky part though is that EFS's can have parameters that aren't necessarily numbers, maybe there are a value "Open", "High", ... Things other than a number. Maybe the parameter falls within some range (> 0 and < 10). Some constraints need to be defined.

My thoughts on defining the constraints or possibilities would be to add some stuff to preMain().

With the below sample, we can construct a smart optimize prompter. It would automatically create a dialog (dynamically)with editboxes, checkboxes, comboboxes/lists as needed.

function preMain() {
    var o1 = new ParameterNumericOption("nLength");
    var o2 = new ParameterBooleanOption("bDoSomething");
    var o3 = new ParameterStringOption("sSomethingElse");

    o1.setLowerLimit(0);
    o1.setUpperLimit(100);
    o1.setDefault(10);
    o1.setOtherOptions(...);

    o3.addOption("Open", true ); // true means default selection
    o3.addOption("High");
    o3.addOption("Low");
    o3.addOption("Close");
}

function main(nLength, bDoSomething, sSomethingElse) {
    ...
} 

Not only could the above be tied to the optimizer/backtester. But, we could better improve the edit studies for formulas. Right now it's very generic. The above would add a level of parameter checking, etc...

Now, if I know Dion, he'll want to break this concept out of preMain into some other builtin function. Maybe

function getParameterInfo(nParameterNumber, sParameterName) {
    if(sParameterName == "nLength") {
        var o = new ParameterNumericOption();
        ...
        return o;
    }

    // Or you could do it this way.
    if(nParameterName == 1) {
        var o = new ParameterNumericOption();
        ...
        return o;
    }
} 

So there's a bunch of ideas. Let's discussl

[gspiker]

Matt said:
[QUOTE]
With the below sample, we can construct a smart optimize prompter. It would automatically create a dialog (dynamically)with editboxes, checkboxes, comboboxes/lists as needed.


code:--------------------------------------------------------------------------------
(Code example deleted)
--------------------------------------------------------------------------------

Not only could the above be tied to the optimizer/backtester. But, we could better improve the edit studies for formulas. Right now it's very generic. The above would add a level of parameter checking, etc...
[QUOTE]

OK, I like that idea, though I still think a majority of your users don't want to learn to code efs to do backtesting, but let's just focus on those of us who don't mind doing so, for now.

I esp. like that you can do range checking and parameter verification (something most .efs study's are pretty weak on right now).

The Problem with defining things like strings, I don't see really being a problem. Why not allow the user to define an array of strings, which you can walk through with the backtester. For example:

My0Params = new ParamterStringArray(4);
My0Params[0] = "Open";
My0Params[1] = "Close";
My0Params[2] = "High";
My0Params[3] = "Low";

This is cleaner I think.

If you do optimization like this (OK, I know you said to ignore this for now, but...), you will have to allow preMain to also influence the name of the backtest file and directory. The autogenerated stuff currently in place is nice - but will not have the scope needed to tell you what the parameters are set to on this run of the test.

Also, if you do implement something like the above, maybe you could also implement a way in preMain() to define an text string to appear in the Edit Study dialog box rather than just variable names. The problem is, variable names may not mean much to users - and if the .efs is encrypted there is no way for even a knowledgeable user to look up what the parameters mean.

For example:

setInputParamName("Number of bars for the moving average", 0);
setInputParamName("Set Type of MA (Must be one of EMA or SMA)", 1);

would procude those strings for Paramter 0 and 1 in the Edit Formula box for this formula rather than:

nInputLnegth
MATYPE

Clearly one is more meaningful than the other.

[mbriand]

Backtesting / Optimizer - OK but first fix bugs

Hello Matt & Garth,

I am ok with the idea of optimizer you have but would like to add the following :

- The optimizer should be able to use a classical algorithm of optimization when you want to optimize more than 1 variable (ie the thing that can take hours on your PC)

- or use Monte Carlo, kind of dichotomy approach.

It would be good also that the EFS editor colors the special optimization variables in a different color so it can be easily find in the code...

BUT...First let us fix the backtester bugs...I have sent many emails saying that the time analysis is buggy but still can reproduce the bugs I found 1 month ago in the 7.2 prebeta...
As you know it is impossible to be more than 100% in the market, with current optimizer it is...It also doesn't report properly the time between trades, can't report properly when you backtest on 180 ticks (will say 180 minutes), doesn't implement a bar approach that would satisfy daytraders (see my strategy.MIDBAR approach), doesn't let you change the monetary symbol to something different than USD, doesn't let you fix the point value for futures and currently eSignal provide so few bars for backtesting (I am still waiting for at least 1 year 1mn bar history or a lot more like Qcharts) that thinking about optimizer right now is a bit too early...

Those little bugs don't need lots of time to be fixed so I still hope next prebeta will see an improvement on this subject so backtester can give us more reliable results.

I also support the other comments about making the backtester v2.0 able to support a list of securities, or a list of system...Or a portfolio of systems acting on different securities each, something Rina provides for Tradestation for example.