Using Custom Backtest to find Min/Max MAE/MFE per Loosing/Winning Trades

xterminator · October 27, 2019, 7:42am

I am trying to use a Custom backtest procedure and output Min/Max MAE/MFE per loosing and winning trades. My idea is to get the nature of loosing trades and determine stop points.
The strategy is intraday based. So my estimate was that I would get the above values during the day since it is running on 1min data.

Here is my backtest proc:

SetCustomBacktestProc("");
if (Status("action") == actionPortfolio) {
	bo = GetBacktesterObject();
	bo.Backtest();
	winningMAE[0] = 0;
	winningMFE[0] = 0;
	loosingMAE[0] = 0;
	loosingMFE[0] = 0;
	wi = 0;
	li = 0;
	for (trade=bo.GetFirstTrade(); trade; trade=bo.GetNextTrade()) {
		if (trade.GetPercentProfit() > 0) {
			winningMAE[wi] = trade.GetMAE();
			winningMFE[wi] = trade.GetMFE();
			wi++;
		}
		else {
			loosingMAE[li] = trade.GetMAE();
			loosingMFE[li] = trade.GetMFE();
			li++;
		}
	}
	--wi;
	--li;
	winningMAE = Sort(winningMAE, first=0, last=wi);
	winningMFE = Sort(winningMFE, first=0, last=wi);
	loosingMAE = Sort(loosingMAE, first=0, last=li);
	loosingMFE = Sort(loosingMFE, first=0, last=li);
	bo.AddCustomMetric("Win L", wi);
	bo.AddCustomMetric("los L", li);
	bo.AddCustomMetric("Min Winning MAE", winningMAE[0]);
	bo.AddCustomMetric("Max Winning MAE", winningMAE[wi]);
	bo.AddCustomMetric("Min Winning MFE", winningMFE[0]);
	bo.AddCustomMetric("Max Winning MFE", LastValue(winningMFE));
	bo.AddCustomMetric("Min Loosing MAE", loosingMAE[0]);
	bo.AddCustomMetric("Max Loosing MAE", LastValue(loosingMAE));
	bo.AddCustomMetric("Min Loosing MFE", loosingMFE[0]);
	bo.AddCustomMetric("Max Loosing MFE", LastValue(loosingMFE));
}

I have the following output. And it can be seen that most values are 0 which is not as expected:

I would also like a Mean function but there is only for MA, is there a way to get array means without using custom looping proc?

fxshrat · October 27, 2019, 10:24am

If you want to store to a custom array then rather use Matrices as they are independent from 1-dim array of length BarCount.

m = MA( Close, 20 ); 
Buy = Cross( Close, m );
Sell = Cross( m, Close );
Short = Cover = 0;

/// @link https://forum.amibroker.com/t/using-custom-backtest-to-find-min-max-mae-mfe-per-loosing-winning-trades/15376/2
/// Storing separate winning/losing MAE/MFE of closed trades to 2-dim array
/// And getting separate winning/losing min/max MAE/MFE of closed trades
/// by fxshrat@gmail.com
SetCustomBacktestProc("");
if (Status("action") == actionPortfolio) {
	bo = GetBacktesterObject();
	bo.Backtest();
	
	wi = li = 0;
	for (trade=bo.GetFirstTrade(); trade; trade=bo.GetNextTrade()) {
		if (trade.GetPercentProfit() > 0)
			wi++;
		else
			li++;
	}

	mat1 = Matrix(Max(1,wi), 2);// winning
	mat2 = Matrix(Max(1,li), 2);// losing
	
	wi = li = 0;
	for (trade=bo.GetFirstTrade(); trade; trade=bo.GetNextTrade()) {
		if (trade.GetPercentProfit() > 0) {
			mat1[wi][0] = trade.GetMAE();
			mat1[wi][1] = trade.GetMFE();
			wi++;
		} else {
			mat2[li][0] = trade.GetMAE();
			mat2[li][1] = trade.GetMFE();
			li++;
		}
	}
	
	sort1 = MxSort(mat1, 1);
	sort2 = MxSort(mat2, 1);	
	//_TRACE(MxToString(sort1));
	//_TRACE(MxToString(sort2));
	
	last_wi = MxGetSize(sort1, 0)-1;
	last_li = MxGetSize(sort2, 0)-1;
	
	// Winning ones
	Min_MAE_wi = sort1[0][0];
	Max_MAE_wi = sort1[last_wi][0];	
	Min_MFE_wi = sort1[0][1];
	Max_MFE_wi = sort1[last_wi][1];

	// Losing ones	
	Min_MAE_li = sort2[0][0];
	Max_MAE_li = sort2[last_li][0];	
	Min_MFE_li = sort2[0][1];
	Max_MFE_li = sort2[last_li][1];

	bo.AddCustomMetric("#Winning", wi);
	bo.AddCustomMetric("#Losing", li);
	// Winning ones
	bo.AddCustomMetric("Min Winning MAE", Min_MAE_wi);
	bo.AddCustomMetric("Max Winning MAE", Max_MAE_wi);
	bo.AddCustomMetric("Min Winning MFE", Min_MFE_wi);
	bo.AddCustomMetric("Max Winning MFE", Max_MFE_wi);
	// Losing ones	
	bo.AddCustomMetric("Min Loosing MAE", Min_MAE_li);
	bo.AddCustomMetric("Max Loosing MAE", Max_MAE_li);
	bo.AddCustomMetric("Min Loosing MFE", Min_MFE_li);
	bo.AddCustomMetric("Max Loosing MFE", Max_MFE_li);
}

Stored (sorted) winning trades array

stored (sorted) losing trades array

If you do not want to store results to array then you do not need Matrix usage at all to get min/max MAE/MFE. You would just need Min/Max functions.

/// @link https://forum.amibroker.com/t/using-custom-backtest-to-find-min-max-mae-mfe-per-loosing-winning-trades/15376/2
/// Getting separate winning/losing min/max MAE/MFE of closed trades
/// by fxshrat@gmail.com
SetCustomBacktestProc("");
if (Status("action") == actionPortfolio) {

	bo = GetBacktesterObject();
	bo.Backtest();
	
	wi = li = 0;
	Min_MAE_wi = Min_MFE_wi = Min_MAE_li = Min_MFE_li = 1e9;
	Max_MAE_wi = Max_MFE_wi = Max_MAE_li = Max_MFE_li = -1e9;	
	for (trade=bo.GetFirstTrade(); trade; trade=bo.GetNextTrade()) {
		if (trade.GetPercentProfit() > 0) {
			Min_MAE_wi = Min(Min_MAE_wi, trade.GetMAE());
			Min_MFE_wi = Min(Min_MFE_wi, trade.GetMFE());
			Max_MAE_wi = Max(Max_MAE_wi, trade.GetMAE());
			Max_MFE_wi = Max(Max_MFE_wi, trade.GetMFE());
			wi++;
		} else {
			Min_MAE_li = Min(Min_MAE_li, trade.GetMAE());
			Min_MFE_li = Min(Min_MFE_li, trade.GetMFE());
			Max_MAE_li = Max(Max_MAE_li, trade.GetMAE());
			Max_MFE_li = Max(Max_MFE_li, trade.GetMFE());
			li++;
		}
	}

	bo.AddCustomMetric("#Winning", wi);
	bo.AddCustomMetric("#Losing", li);
	// Winning ones
	bo.AddCustomMetric("Min Winning MAE", Min_MAE_wi);
	bo.AddCustomMetric("Max Winning MAE", Max_MAE_wi);
	bo.AddCustomMetric("Min Winning MFE", Min_MFE_wi);
	bo.AddCustomMetric("Max Winning MFE", Max_MFE_wi);
	// Losing ones	
	bo.AddCustomMetric("Min Loosing MAE", Min_MAE_li);
	bo.AddCustomMetric("Max Loosing MAE", Max_MAE_li);
	bo.AddCustomMetric("Min Loosing MFE", Min_MFE_li);
	bo.AddCustomMetric("Max Loosing MFE", Max_MFE_li);
}

To get mean of MAE/MFE just sum up within loop and then divide by wi/li.

xterminator · October 27, 2019, 4:00pm

Hey fkshrat, thanks again for your solution.
I also want to understand what I am doing wrong.
So when we create any array in AFL, it automatically becomes BarCount - 1 length?
And if yes, then here I was using wi & li index variables to access the first(0) and last(wi/li) elements. So I am failing to understand what I was doing wrong in my solution?

I hope to improve my understanding. Regards.

Cougar · October 27, 2019, 4:24pm

@xterminator,

Before anything else, you seriously need to read (yes READ) this: Understanding how AFL works.

DO NOT jump to advanced concepts without understanding basics. Three days back you had trouble understanding this! Today you're discussing on "Custom Backtest".

What fxshrat explained, you would've easily understood if your basics were clear? Please check whether "Help Vampire" syndrome is getting heavy on you!

Read: How do I debug my formula?

xterminator · October 27, 2019, 5:02pm

Thanks again for you reply.
I see that but nowhere it mentions that what will be the initial size of the array when one is declared. I am talking about custom arrays. Since it is not mentioned, I assume that custom array length will be the same as the size of data I insert.

Then fxshrat wrote that: "Use matrix to avoid BarCount -1 length limitations".
So even if all custom arrays are automatically made to BarCount - 1 (after filling the remaining data points with Null) I still cannot figure out why my code went wrong since I am ONLY relying on the maximum data inserted index to access value.

I also appreciate such great solutions given pre-made to me but I would like to know the internals myself.

fxshrat · October 27, 2019, 5:40pm

@xterminator,

once again... in AFL all 1-dim arrays have length BarCount.

The article of "Understanding how AFL works" mentions it already:

All array indices in AFL are zero-based, i.e. counting bars starting from bar 0 (oldest). The latest (newest) bar has an index of (BarCount-1). The 10-element array will have BarCount = 10 and indices going from 0 to 9 as shown below

In your case also e.g. WinningMAE is not of length wi but is of length BarCount as well as all of your other arrays of your first post.

Then... you iterate trade list (from first closed trade to last closed trade) but not from 0 to Barcount-1.

wi/li may be length BarCount but it is highly unlikely depending on number of trades and on if statement being true.

So LastValue() function will access last element of array of length Barcount. Lastvalue does access element at bar-index BarCount-1 but not at indexes wi/li.

Also again.... you do not need to create any array to get the output you are after.

m = MA( Close, 20 ); 
Buy = Cross( Close, m );
Sell = Cross( m, Close );
Short = Cover = 0;

/// @link https://forum.amibroker.com/t/using-custom-backtest-to-find-min-max-mae-mfe-per-loosing-winning-trades/15376/2
/// Getting separate winning/losing min/max MAE/MFE of closed trades
/// And getting separate mean of winning/losing MAE/MFE
/// by fxshrat@gmail.com
SetCustomBacktestProc("");
if (Status("action") == actionPortfolio) {

	bo = GetBacktesterObject();
	bo.Backtest();
	
	wi = li = 0;
	csMAE_wi = csMFE_wi = csMAE_li = csMFE_li = 0;
	Min_MAE_wi = Min_MFE_wi = Min_MAE_li = Min_MFE_li = 1e9;
	Max_MAE_wi = Max_MFE_wi = Max_MAE_li = Max_MFE_li = -1e9;
	for (trade=bo.GetFirstTrade(); trade; trade=bo.GetNextTrade()) {
		MAE = trade.GetMAE();
		MFE = trade.GetMFE();
		if (trade.GetPercentProfit() > 0) {
			Min_MAE_wi = Min(Min_MAE_wi, MAE);
			Min_MFE_wi = Min(Min_MFE_wi, MFE);
			Max_MAE_wi = Max(Max_MAE_wi, MAE);
			Max_MFE_wi = Max(Max_MFE_wi, MFE);
			csMAE_wi += trade.GetMAE();
			csMFE_wi += trade.GetMFE();
			wi++;
		} else {
			Min_MAE_li = Min(Min_MAE_li, MAE);
			Min_MFE_li = Min(Min_MFE_li, MFE);
			Max_MAE_li = Max(Max_MAE_li, MAE);
			Max_MFE_li = Max(Max_MFE_li, MFE);
			csMAE_li += MAE;
			csMFE_li += MFE;
			li++;
		}
	}

	bo.AddCustomMetric("#Winning", wi);
	bo.AddCustomMetric("#Losing", li);
	bo.AddCustomMetric("");
	// Winning ones
	bo.AddCustomMetric("Min Winning MAE", Min_MAE_wi);
	bo.AddCustomMetric("Max Winning MAE", Max_MAE_wi);
	bo.AddCustomMetric("Min Winning MFE", Min_MFE_wi);
	bo.AddCustomMetric("Max Winning MFE", Max_MFE_wi);
	bo.AddCustomMetric("");
	bo.AddCustomMetric("Mean Winning MAE", csMAE_wi/wi);
	bo.AddCustomMetric("Mean Winning MFE", csMFE_wi/wi);
	bo.AddCustomMetric("");
	// Losing ones	
	bo.AddCustomMetric("Min Loosing MAE", Min_MAE_li);
	bo.AddCustomMetric("Max Loosing MAE", Max_MAE_li);
	bo.AddCustomMetric("Min Loosing MFE", Min_MFE_li);
	bo.AddCustomMetric("Max Loosing MFE", Max_MFE_li);	
	bo.AddCustomMetric("");
	bo.AddCustomMetric("Mean Loosing MAE", csMAE_li/li);
	bo.AddCustomMetric("Mean Loosing MFE", csMFE_li/li);	
}

xterminator · October 27, 2019, 10:27pm

I used the storing version and added mean calculations. Just my little enhancement:

function calcMean(array, total) {
	s = 0;
	for (i = 0; i <= total; i++)
		s += array[i];
	return s/total;
};


SetCustomBacktestProc("");
if (Status("action") == actionPortfolio) {
	bo = GetBacktesterObject();
	bo.Backtest();
	
	wi = li = 0;
	for (trade=bo.GetFirstTrade(); trade; trade=bo.GetNextTrade()) {
		if (trade.GetPercentProfit() > 0)
			wi++;
		else
			li++;
	}

	mat1 = Matrix(Max(1,wi), 2);// winning
	mat2 = Matrix(Max(1,li), 2);// losing
	
	wi = li = 0;
	for (trade=bo.GetFirstTrade(); trade; trade=bo.GetNextTrade()) {
		if (trade.GetPercentProfit() > 0) {
			mat1[wi][0] = trade.GetMAE();
			mat1[wi][1] = trade.GetMFE();
			wi++;
		} else {
			mat2[li][0] = trade.GetMAE();
			mat2[li][1] = trade.GetMFE();
			li++;
		}
	}
	
	sort1 = MxSort(mat1, 1);
	sort2 = MxSort(mat2, 1);	
	//_TRACE(MxToString(sort1));
	//_TRACE(MxToString(sort2));
	
	last_wi = MxGetSize(sort1, 0)-1;
	last_li = MxGetSize(sort2, 0)-1;
	
	// Winning ones
	Min_MAE_wi = sort1[0][0];
	Max_MAE_wi = sort1[last_wi][0];	
	Min_MFE_wi = sort1[0][1];
	Max_MFE_wi = sort1[last_wi][1];

	// Losing ones	
	Min_MAE_li = sort2[0][0];
	Max_MAE_li = sort2[last_li][0];	
	Min_MFE_li = sort2[0][1];
	Max_MFE_li = sort2[last_li][1];

	// Winning ones
	bo.AddCustomMetric("Min Winning MAE", Min_MAE_wi);
	bo.AddCustomMetric("Max Winning MAE", Max_MAE_wi);
	bo.AddCustomMetric("Mean Winning MAE", calcMean(MxGetBlock(sort1, 0, last_wi, 0, 0, True), last_wi));
	bo.AddCustomMetric("Min Winning MFE", Min_MFE_wi);
	bo.AddCustomMetric("Max Winning MFE", Max_MFE_wi);
	bo.AddCustomMetric("Mean Winning MFE", calcMean(MxGetBlock(sort1, 0, last_wi, 1, 1, True), last_wi));
	// Losing ones	
	bo.AddCustomMetric("Min Loosing MAE", Min_MAE_li);
	bo.AddCustomMetric("Max Loosing MAE", Max_MAE_li);
	bo.AddCustomMetric("Mean Loosing MAE", calcMean(MxGetBlock(sort2, 0, last_li, 0, 0, True), last_li));
	bo.AddCustomMetric("Min Loosing MFE", Min_MFE_li);
	bo.AddCustomMetric("Max Loosing MFE", Max_MFE_li);
	bo.AddCustomMetric("Mean Loosing MFE", calcMean(MxGetBlock(sort2, 0, last_li, 1, 1, True), last_li));
}

fxshrat · October 27, 2019, 11:40pm

You do not need to apply looping at all to create mean of matrix vectors.
You just need Matrix multiplication using @ operator.
Sum of elements in column vectors of matrix X ->
Row vector V of ones (with column dimension == row dimension of matrix X) @ matrix X.
Then division by rownum X.

V @ X / RownumX

Results in row vector (1xn) if X is m x n matrix and n > 1. Otherwise matrix of size 1x1.

Also I would appreciate if you do not remove links and creator of working code since AFAICS it wasn't you who got it done properly but actually (as far as my memory serves me right) it has been my time spent for complete rewrite.

m = MA( Close, 20 ); 
Buy = Cross( Close, m );
Sell = Cross( m, Close );
Short = Cover = 0;

/// original code 
/// @link https://forum.amibroker.com/t/using-custom-backtest-to-find-min-max-mae-mfe-per-loosing-winning-trades/15376
/// fixed rewrite:
/// @link https://forum.amibroker.com/t/using-custom-backtest-to-find-min-max-mae-mfe-per-loosing-winning-trades/15376/8
/// Storing separate winning losing MAE/MFE of closed trades to 2-dim array
/// And getting separate winning/losing min/max MAE/MFE of closed trades
/// And getting separate mean of winning/losing MAE/MFE
/// by fxshrat@gmail.com
SetCustomBacktestProc("");
if (Status("action") == actionPortfolio) {
	bo = GetBacktesterObject();
	bo.Backtest();
	
	wi = li = 0;
	for (trade=bo.GetFirstTrade(); trade; trade=bo.GetNextTrade()) {
		if (trade.GetPercentProfit() > 0)	wi++;
		else								li++;
	}

	mat1 = Matrix(Max(1,wi), 2);// winning
	mat2 = Matrix(Max(1,li), 2);// losing
	
	wi = li = 0;
	for (trade=bo.GetFirstTrade(); trade; trade=bo.GetNextTrade()) {
		MAE = trade.GetMAE();
		MFE = trade.GetMFE();
		if (trade.GetPercentProfit() > 0) {
			mat1[wi][0] = MAE;
			mat1[wi][1] = MFE;
			wi++;
		} else {
			mat2[li][0] = MAE;
			mat2[li][1] = MFE;
			li++;
		}
	}
	
	sort1 = MxSort(mat1, 1);
	sort2 = MxSort(mat2, 1);	
	//_TRACE(MxToString(sort1));
	//_TRACE(MxToString(sort2));	
	
	rownum1 = MxGetSize(sort1, 0);
	rownum2 = MxGetSize(sort2, 0);
	last_wi = rownum1-1;
	last_li = rownum2-1;		
	
	// Winning ones
	Min_MAE_wi = sort1[0][0];
	Max_MAE_wi = sort1[last_wi][0];	
	Min_MFE_wi = sort1[0][1];
	Max_MFE_wi = sort1[last_wi][1];
	// Mean winning
	cs_mat1 = Matrix(1, rownum1, 1) @ sort1;
	mMAE_wi = cs_mat1[0][0]/rownum1;
	mMFE_wi = cs_mat1[0][1]/rownum1;

	// Losing ones	
	Min_MAE_li = sort2[0][0];
	Max_MAE_li = sort2[last_li][0];	
	Min_MFE_li = sort2[0][1];
	Max_MFE_li = sort2[last_li][1];
	// Mean losing
	cs_mat2 = Matrix(1, rownum2, 1) @ sort2;	
	mMAE_li = cs_mat2[0][0]/rownum2;
	mMFE_li = cs_mat2[0][1]/rownum2;

	bo.AddCustomMetric("#Winning", wi);
	bo.AddCustomMetric("#Losing", li);
	bo.AddCustomMetric("");
	// Winning ones
	bo.AddCustomMetric("Min Winning MAE", Min_MAE_wi);
	bo.AddCustomMetric("Max Winning MAE", Max_MAE_wi);
	bo.AddCustomMetric("Mean Winning MAE", mMAE_wi);
	bo.AddCustomMetric("");	
	bo.AddCustomMetric("Min Winning MFE", Min_MFE_wi);
	bo.AddCustomMetric("Max Winning MFE", Max_MFE_wi);
	bo.AddCustomMetric("Mean Winning MFE", mMFE_wi);
	bo.AddCustomMetric("");
	// Losing ones	
	bo.AddCustomMetric("Min Loosing MAE", Min_MAE_li);
	bo.AddCustomMetric("Max Loosing MAE", Max_MAE_li);
	bo.AddCustomMetric("Mean Loosing MAE", mMAE_li);
	bo.AddCustomMetric("");
	bo.AddCustomMetric("Min Loosing MFE", Min_MFE_li);
	bo.AddCustomMetric("Max Loosing MFE", Max_MFE_li);
	bo.AddCustomMetric("Mean Loosing MFE", mMFE_li);	
}

xterminator · October 27, 2019, 11:41pm

Hey, my removing link wasn't intentional. I pasted your code in a long AFL system of mine and I forgot to copy it back after making the modifications.
Thanks again for the suggestions.

Cougar · October 28, 2019, 8:39am

Ha ha ha.... Yeah, everyone can see the future....

Each and every question that you ask, to answer that, someone has to spend a considerable amount of time to code, test, experiment and post. So, mostly nothing is "prep-made".

If you do not respect other's time, don't expect that in reciprocal?

FYI, meaning of a term.