Backtester and Exploration results mismatch

Hello guys, please don't kill me, I know that Backtester and Exploration are different, so Exploration is raw and some backtester commands are not available... But I'm facing an issue that seems reverting the point of view. Seems to be an issue more related to parameters than to commands and It appears that exploration results are more accurate than the backtester one. I'm asking for help because I really can't figure out the following:

I've coded a trading system adding some exploration's columns for debugging tasks and I've found that in certain parameter's conditions the exploration results differs from the backtester ones, due the fact that some right buy signals are ignored by the backtester (or at least this seems to me). More curiously plotting the chart all the signal are plotted correctly.

The code to recreate the issue is the following:

//	logicTest.afl

//	Backtester parameters
SetOption("InitialEquity",100000);
SetOption("AllowPositionShrinking", True);
maxPos = 10;
SetOption("MaxOpenPositions", maxPos);
SetOption("CommissionMode", 1);
SetOption("CommissionAmount", 0);
SetPositionSize(10000,spsValue);
SetTradeDelays(1,0,0,0);
BuyPrice = O;
SellPrice = C;

//	Trading system parameters
gapType = ParamList("gap type (day 0)", "UP|DOWN");
barDay0 = ParamList("day 0 bar type", "BULL|BEAR|DOJI");
barDayP = ParamList("day -1 bar type", "ANYKIND|BULL|BEAR|DOJI");
trdDuration = Param("Trade duration", 5, 1,30, 1); 

//	evaluate the bars
barDay0T = IIf(C>O, 1, IIf(C<O,-1,0));
barDayPT = Ref(barDay0T,-1);

//	constraints
condGapDay0 = IIf(gapType == "UP", GapUp(), GapDown());

condBarDay0 = ((barDay0 == "BULL") AND (barDay0T == 1)) OR
				((barDay0 == "BEAR") AND (barDay0T == -1)) OR
				((barDay0 == "DOJI") AND (barDay0T == 0));

condBarDayP = ((barDayP == "BULL") AND (barDayPT == 1)) OR
			((barDayP == "BEAR") AND (barDayPT == -1)) OR
			((barDayP == "DOJI") AND (barDayPT == 0)) OR
			(barDayP == "ANYKIND") ;

//	signals
Buy = condGapDay0 AND condBarDay0 AND condBarDayP;
Sell = 0;
ApplyStop(stopTypeNBar, stopModeBars, trdDuration-1);

// 	exploration columns
AddColumn(O, "open", 1.5);
AddColumn(H, "high", 1.5);
AddColumn(L, "low", 1.5);
AddColumn(C, "close", 1.5);
AddColumn(barDay0T, "Tipo Barra", 1.0);
AddColumn(barDayPT, "Tipo BarraP", 1.0);
AddColumn(condGapDay0, "CondGap", 1.0, colorBlack,IIf(condGapDay0,colorGreen,colorWhite));
AddColumn(condBarDay0, "condBarDay0", 1.0, colorBlack,IIf(condBarDay0,colorGreen,colorWhite));
AddColumn(condBarDayP, "condBarDayP", 1.0, colorBlack,IIf(condBarDayP,colorGreen,colorWhite));
AddColumn(BUY, "BUY SIGNAL", 1.0, colorBlack,IIf(buy,colorRed,colorWhite));

filter = Buy;
//Filter = True;

//	plots
barsColor = IIf(condGapDay0, colorRed, colorBlack);
PlotOHLC(O,H,L,C,"",barsColor,styleBar,Null,Null,0,-1,3);
SetChartOptions(0, chartShowDates);
PlotShapes(shapeUpArrow,colorGreen, 0, BuyPrice*Buy, -30, 1);

The example uses the daily time series of a2a present in Yahoo Finance (ticker = A2A.MI) . The backtest and exploration period spans between 01/01/2010 to 31/12/2019

The logic is very simple: when a gap occurs check the kind of the current bar and the previous one. If all they match the criterions (setted by paramlist) then buy the next day at open and keep the position for a certain number of bars.

Criterions are to checking the kinds of bars: ex. BULL bar means that close price is higher than open. etc..

THE ISSUE:
If I set the parameters : search for GAP UP, today bar is BULLISH and yesterday bar is BEARISH, exploration and backtester give the same results : 2 trades.

BUT if I set the parameters : search for GAP UP, today bar is BULLISH and yesterday bar is also BULLISH, exploration and backtester give the two different results : exploration 37 trades , backtester 31.
For example the exploration says that a buy signal occurs the 15th of march, plotting correctly shows the setup and the arrow on the 16th but the backtester don't take that trade.

Why the results differs? I've asked the exploration to plot the buy signals, and they are the same that are supposed to be used by the backtester.

Any HELP or insight is more than welcome!!

Marco

Screenshots:

CORRECT:
OK - 2 trades

WRONG:
WRONG - different results

Plotted arrow not traded:
Schermata 2021-04-10 alle 18.04.26

Solution 1: Users' Guide Portfolio-level back testing - please read "Backtest modes" and look at the picture showing how signals are matched by backtester.

Solution 2: Please see: Scan vs Exploration vs Backtest - summary of differences

Backtest-only functions like SetTradeDelays, ApplyStop, many SetOption calls have absolutely NO effect on exploration

Yet you have this

SetTradeDelays(1,0,0,0);

SetTradeDelays is backtester only function and has no effect on exploration. If you use it you are deliberately making exploration generating different signals than backtester. Backtester buy signal will come one bar LATER because of SetTradeDelays call.

*Disclaimer: pointing the user to read existing documentation is NOT insult. It is meant as HELP. *

2 Likes

Thank you Tomasz,
first of all I don't feel insulted at all. I consider your pointing an help.
I have to say that before posting I've done some research into the forum, I've read some other posts and tried to figure out how they can help me, but I didn't found a way. Then I've raw analyzed the time serie using excel finding that the exploration results were correct and the backtester ones were not. For this reason I've decided to posted my issue, that could seems redundant, because I felt it was a different kind of issue.

I've finally found the solution thanks to your suggestion to re study the documentation related to how the backtest modes works.

Solution for me is to add the line

SetBacktestMode(backtestRegularRawMulti);

So it was not an issue related to any unsupported exploration commands (such as SetTradeDelays) but it was related the wrong backtester mode I've used (the default one).

I hope this will help someone else in the future.

Thanks again
Marco

1 Like

Are you sure you want to open multiple positions on the same symbol ? If not, I would recommend you to use backtestRegularRaw instead.

Yes, the trading system I've written is part of a statistical research on gaps, so it is not intended for trading real money. It helps me for the evaluation of metrics and other analysis. Thanks for the hint anyway
Marco

1 Like