Individual optimization results NOT different than backtester's

Hello,
in my optimizable code I have this:

gapCases = Null;
gapCasesOpti = Optimize( "gapCasesOpti", 3, 1, 3, 1 );
gapAmountOpti = Optimize( "gapAmountOpti", 4.5, 0, 5, 0.50 );
switch( gapCasesOpti )
{
case 1:
    gapCases = True;
    break;
case 2:
    gapCases = todayOpen < yesterdayClose - gapAmountOpti;
    break;
case 3:
    gapCases = todayOpen > yesterdayClose + gapAmountOpti;
    break;
default:
	_TRACE("Default case");
    break;
}

Not sure if this kind of optimization is "legit" which could be the source of my problem.
Anyway when I run the individual optimization, here is the net profit for a particular combination of optimizable parameters:

opti2

When I add corresponding parameters from optimization as default parameters, the backtesting result is very different (net profit 385).

What could cause difference between Backtesting results, and optimization results when corresponding optimization values are added as default values to be used in backtester?

The code snippet appears to be valid, but as always it's hard to provide feedback on why your code doesn't work when you don't provide the code. One thing that could cause issues is if you're using Static Variables and not deleting them at the end (or beginning) of each backtest. That could potentially allow you to "reuse" a static variable from a previous backtest when you didn't mean to.

If you're not able to debug the problem using the Debugger or _TRACE statements, then my suggestion is to simplify your code as much as possible while still being able to demonstrate the problem. Post the simplified version and someone will likely be able to offer some suggestions.

1 Like

OK, here is a full code. It may not make perfect sense, but it is a simplified version where the problem can be tested.

Opimization result:

opti4

Backtest result with same parameters: net profit: 486.32

currencySymbol = "$";
systemName = ParamStr( "System Name", "~~MicroTarget_230919" );
initialEquity = Param( "Initial Equity", 1000, 10, 10000, 10 );
commissionAmount = Param( "Commission Amount", 0.47, 0, 10, 0.1 );
positionAmount = Param( "Position Amount", 1, 1, 100, 1 );


SetOption("AllowSameBarExit", True );
SetOption("ActivateStopsImmediately", True );
SetOption( "InitialEquity"  , initialEquity );
SetPositionSize( positionAmount, spsShares ); // Force position size to use 1 contract only
SetOption( "CommissionMode" , 3 ); // Mode 3 = perShareContract
SetOption( "CommissionAmount"  , commissionAmount );
SetOption( "ExtraColumnsLocation", 1 );
SetOption("MaxOpenPositions", 2);
SetOption("MaxOpenLong", 1 );
SetOption("MaxOpenShort", 1 );
SetTradeDelays( 0, 0, 0, 0 );
//#include_once <_Backtesters/defaultCBt.afl> // Full heavy CBT

/**************************************************/
/*	   SYSTEM VARIABLES
/**************************************************/
timeNow = TimeNum();
dateNow = DateNum();
firstBarTime = 093000;
lastTradeTime = 154000;
firstBarOfDay = TimeNum() == firstBarTime;
secondBarOfDay = BarsSince(firstBarOfDay) == 1;
currentDayHigh = Ref(ValueWhen( timeNow, HighestSince( firstBarOfDay, High ) ), -1);
currentDayLow = Ref(ValueWhen( timeNow, LowestSince( firstBarOfDay, Low ) ), -1);
yesterdayClose = TimeFrameGetPrice( "C", inDaily, -1 );
todayOpen = TimeFrameGetPrice( "O", inDaily );


/****************************************************************************************************/
/*	GAPS	                                    
/****************************************************************************************************/
gapCases = Null;
gapCasesOpti = Optimize( "gapCasesOpti", 3, 1, 3, 1 );
gapAmountOpti = Optimize( "gapAmountOpti", 4, 0, 5, 0.50 );
switch( gapCasesOpti )
{
case 1:
    gapCases = True;
    break;
case 2:
    gapCases = todayOpen < yesterdayClose - gapAmountOpti;
    break;
case 3:
    gapCases = todayOpen > yesterdayClose + gapAmountOpti;
    break;
default:
	_TRACE("Default case");
    break;
}




/****************************************************************************************************/
/*	ENTRIES	                                    
/****************************************************************************************************/
myEntries = Null;
myEntriesOpti = Optimize( "myEntries", 2, 1, 2, 1 );

switch( myEntriesOpti )
{
case 1:
    myEntries = True;
    break;
case 2:
	// Open + Fraction of Average Daily Range
    myEntries = C > todayOpen + 1;
    break;
default:
	_TRACE("Default case");
    break;
}
/****************************************************************************************************/
/*	EXITS	                                    
/****************************************************************************************************/


BuyPrice = C;
Buy = myEntries
	AND gapCases;

exitBeforeDayEnd = Cross( timeNow, lastTradeTime );

// Exit Longs
SellPrice = C;
Sell = exitBeforeDayEnd;

When posting this kind of post I always fear the error is so obvious I will feel ridiculous :confused:

You are using Param* functions so if you set to non default values in optimization and then you open additional analysis window to do backtest make sure the same values are set as params may reset to default. So rather use hard coded values instead Param.

Or check via Trace to be sure same ones are used

systemName = ParamStr( "System Name", "~~MicroTarget_230919" );
initialEquity = Param( "Initial Equity", 1000, 10, 10000, 10 );
commissionAmount = Param( "Commission Amount", 0.47, 0, 10, 0.1 );
positionAmount = Param( "Position Amount", 1, 1, 100, 1 );

_TRACEF("%s, %g, %g, %g", systemName, initialEquity, commissionAmount, positionAmount);

As aside below variables are unused variables... but still first argument of ValueWhen does not return true/false as TimeNow (which just holds TimeNum() array) is always true except for time being 00:00:00 (TimeNum() -> zero).

currentDayHigh = Ref(ValueWhen( timeNow, HighestSince( firstBarOfDay, High ) ), -1);
currentDayLow = Ref(ValueWhen( timeNow, LowestSince( firstBarOfDay, Low ) ), -1);
2 Likes

Thank you for your answer and your remark about ValueWhen.
I hardcoded the params as you suggest, but it doesn't change the difference.

With the above code, do you also notice difference between backtest and optimization when they both use same default opti parameters?

Read the Knowledge Base:
http://www.amibroker.com/kb/2015/01/02/using-optimum-parameter-values-in-backtesting/

remove Param()s and just use _TRACE as advised by @fxshrat to display actual REAL values of variables that are being used because as it is now, you don't know what you are doing: How do I debug my formula?

Hi,
I read the kb article and debug. I think I set the backtest default parameters properly and I removed the parameters.

I noticed the divergence in results from backtest and optimization, happen only when using the "individual optimize". When using the regular single threaded optimize results matches.

Here is full code:

currencySymbol = "$";
systemName = "~~MicroTarget_230919";
initialEquity = 1000;
commissionAmount = 0.47;
positionAmount = 1;


SetOption("AllowSameBarExit", True );
SetOption("ActivateStopsImmediately", True );
SetOption( "InitialEquity"  , initialEquity );
SetPositionSize( positionAmount, spsShares ); // Force position size to use 1 contract only
SetOption( "CommissionMode" , 3 ); // Mode 3 = perShareContract
SetOption( "CommissionAmount"  , commissionAmount );
SetOption( "ExtraColumnsLocation", 1 );
SetOption("MaxOpenPositions", 2);
SetOption("MaxOpenLong", 1 );
SetOption("MaxOpenShort", 1 );
SetTradeDelays( 0, 0, 0, 0 );
//#include_once <_Backtesters/defaultCBt.afl> // Full heavy CBT

/**************************************************/
/*	   SYSTEM VARIABLES
/**************************************************/
timeNow = TimeNum();
dateNow = DateNum();
firstBarTime = 093000;
lastTradeTime = 154000;
firstBarOfDay = TimeNum() == firstBarTime;
secondBarOfDay = BarsSince(firstBarOfDay) == 1;
currentDayHigh = Ref(ValueWhen( timeNow, HighestSince( firstBarOfDay, High ) ), -1);
currentDayLow = Ref(ValueWhen( timeNow, LowestSince( firstBarOfDay, Low ) ), -1);
yesterdayClose = TimeFrameGetPrice( "C", inDaily, -1 );
todayOpen = TimeFrameGetPrice( "O", inDaily );


/****************************************************************************************************/
/*	GAPS	                                    
/****************************************************************************************************/
gapCases = Null;
gapCasesOpti = Optimize( "gapCasesOpti", 3, 1, 3, 1 );
gapAmountOpti = Optimize( "gapAmountOpti", 4, 0, 5, 0.50 );
switch( gapCasesOpti )
{
case 1:
    gapCases = True;
    break;
case 2:
    gapCases = todayOpen < yesterdayClose - gapAmountOpti;
    break;
case 3:
    gapCases = todayOpen > yesterdayClose + gapAmountOpti;
    break;
default:
	_TRACE("Default case");
    break;
}




/****************************************************************************************************/
/*	ENTRIES	                                    
/****************************************************************************************************/
myEntries = Null;
myEntriesOpti = Optimize( "myEntries", 2, 1, 2, 1 );

switch( myEntriesOpti )
{
case 1:
    myEntries = True;
    break;
case 2:
	// Open + Fraction of Average Daily Range
    myEntries = C > todayOpen + 1;
    break;
default:
	_TRACE("Default case");
    break;
}
/****************************************************************************************************/
/*	EXITS	                                    
/****************************************************************************************************/


BuyPrice = C;
Buy = myEntries
	AND gapCases;

exitBeforeDayEnd = Cross( timeNow, lastTradeTime );

// Exit Longs
SellPrice = C;
Sell = exitBeforeDayEnd;


Here is the full context, watch net profit:

Backtest results 194.15 $ net profit:
setupk

Optimization on single thread 194.15 $ net profit, all is fine:
optiregular

Individual optimization on multiple threads net profit: 2185.40 $:
optiindividual

I get different optimization results per optimization parameter combination, with the only thing I change is clicking on "Optimize" or "Individual Optimize".

Why "Single optimize" produces different results than regular Optimize is beyond my comprehension and I would appreciate an explanation please.

I notice in both cases your net % profit is the same. I suspect maybe you have different initial equity settings for the two cases. Have you checked this? Alternatively, you may have different position sizing.

Just click on "Optimize" button even if you test on single symbol. That is the default and the one that should be used.

As to MT individual optimization mode, I have run your code and I am getting same results from MT individual and regular optimization and backtest, but anyway MT individual is marked in Release Notes as experimental - exactly what CPU ( how many cores) you have ?

I have an Intel Quad Core i7 4770HQ.
I suggest you try with the instrument file I sent to support.... this problem may happen with certain data only.

As I wrote: I tested on YOUR data using YOUR formula and YOUR settings and I am getting 100% identical results with both Individual Optimize and Optimize. I have sent you the video documenting both.

I would suggest some stress testing of your hardware as you may have unstable machine / memory (RAM) issues.