MCS optimization target missing in walk-forward testing?

AFL Programming
1

I'm getting a strange error, but having trouble reproducing it consistently.

I'm using MCS stats as an optimization target as described here: https://www.amibroker.com/guide/h_montecarlo.html

I sometimes, but not always, get Error 711, telling me that the target can not be found in the result list. However, see the screenshot where CAR25/DD25 is circled in red, because it is definitely there.

image

At least twice, simply closing AB and restarting it solved the issue. For example, after closing and re-opening AB, I did the exact same test on the same data, and AB didn't give that error.

However, the walk forward results were corrupted. The variable being optimized, MaxPosValPct, is not properly populated in this screenshot. Some of the values are appearing one column to the left, in the 'Probability of breakeven' column:
image

Am I missing something here? Any advice on what to look for next?

2

Your system MUST produce at least one trade, so custom metric is NOT NULL. Otherwise you will get an error message like this.

3

Thanks for responding so fast Tomasz! I truly appreciate your help.

There were between 75-150 trades every year in the walk forward period.

Here's another screenshot, showing the number of trades also:
image

4

These columns show just the WINNERS for in-sample period. They don't represent ALL in-sample steps. You have to watch out for ALL parameter combinations for ALL in-sample steps. They MUST produce non-null values for custom metric if such metric is used as optimization target. In other words, in your case CAR25/DD95 MUST NOT be Null no matter what parameter combination is used.

At minimum protect against Nulls using Null-To-Zero - the Nz() function

bo.AddCustomMetric("MyMetric", Nz( your_value ) );
2 Likes
5

Ok, I'll look at the trade list and also the walk forward periods to ensure there are trades in each one, and that the metrics are being calculated.

Thanks so much again, Tomasz!

6

@PeterD Something else to look for is conditional code around the custom metric.

Avoid this pattern:

if (condition)
    bo.AddCustomMetric("MyMetric", myValue);

If a valid metric value cannot be calculated for some reason, do this instead:

if (condition)
    bo.AddCustomMetric("MyMetric", myValue);
else
    bo.AddCustomMetric("MyMetric", defaultValue);
3 Likes
7

Thanks, Steve! That's a great idea.