Error during long running optimization

Hi,

I was running a long running optimization (~ 5 days - I need to get a faster computer!).

Relevant excerpts from the code:

// LEAVE THESE OPTIONS COMMENTED OUT UNLESS YOU FIRST READ  
// https://www.amibroker.com/guide/h_optimization.html  
// However, running one of these non-exhaustive optimizations may allow  
// optimization of more parameters in combination instead of in small groups.  
  
OptimizerSetEngine("trib");  
OptimizerSetOption("Runs", 5); 			// default value 
OptimizerSetOption("MaxEval", 20000); 	// max number of evaluations  

and

// ROC Lookback  
ROCShortLookback		= Optimize("ROCShort",  2, 1, 12, 1);  
ROCLongLookback			= Optimize("ROCLong",  28, 8, 52, 1);  

and

ROCShort    = ROC(C,ROCShortLookback);  
ROCLong     = ROC(C,ROCLongLookback);  

This is the error message from the optimization window:

image

Column 19 of line 265 is the Close (C) array in the ROCShortLookback line. I guess that implies that Close was Null or negative?

What is the best way to trap for this error? Should I wrap Close and ROCShortLookback in the Nz function? It would be cool if there was a setting that said "ignore errors" during optimization. A bit like Powershell's ErrorAction = Ignore or SilentlyIgnore. Or else Try/Catch/Except functionality in AFL.

I also get "weird" results sometimes, such as NetProfit = -298409998935111.38. I assume that's just from dividing by some really small number? I just ignore those results, since I don't have that much money to lose :wink:

Finally, I would love to have a better understanding of how the non-exhaustive optimization engines work: cmae, spso, and trib. I have read https://www.amibroker.com/guide/h_optimization.html as well as the included links. In layman's terms, my guess is these engines use statistical algorithms to select a random set of parameters within the optimization parameter set, run the backtests, then use statistics/machine learning/etc. to pick another set of parameters to zero in on the objective function (say CAR/MDD or NetProfit).

My empirical evidence is I get better results when I run the non-exhaustive optimization rather than doing an exhaustive optimization "in chunks". IOW, while the exhaustive optimization will test every possibility in the optimization data set, optimizing the entirety of the optimization data set together using the non-exhaustive optimization gives me better results. I also seem to get better results using trib. Note I'm not favoring trib over the other engines, just stating my personal experience.

But of course I could be completely wrong on this, thus my questions!

Thanks for your help.

No, range refers to ROC() period (2nd argument).

Are you sure lookback periods are not redefined/adjusted somewhere else in your code?
Simply _Trace() your two lookback variables.
Also do a variable search via your Formula editor.

Anyway you could do this

// ROC Lookback  
ROCShortLookback		= Optimize("ROCShort",  2, 1, 12, 1);  
ROCLongLookback			= Optimize("ROCLong",  28, 8, 52, 1); 

ROCShort    = ROC(C,Max(1,ROCShortLookback));  
ROCLong     = ROC(C,Max(8,ROCLongLookback)); 

And/Or use Exclude variable

// ROC Lookback  
ROCShortLookback		= Optimize("ROCShort",  2, min1 = 1, 12, 1);  
ROCLongLookback			= Optimize("ROCLong",  28, min2 = 8, 52, 1); 

//....

Exclude = ROCShortLookback < min1 OR ROCLongLookback < min2;
1 Like

Thanks @fxshrat for your quick reply!

Those two lines are the only references to ROCShortLookback when I search my code.

So, if the error message is correct and not a "red herring", and if Close cannot be negative or NaN (my universe is stocks), then I don't understand how ROCShortLookback became negative or NaN given the parameters I passed to Optimize?

On first glance, I was tempted to use the Max(1,ROCShortLookback) approach.

However, I just Googled and found http://www.amibroker.com/kb/2015/02/05/using-exclude-statement-to-skip-unwanted-optimization-steps/. It looks like this would really benefit my optimization and hopefully improve performance. I can certainly see where I can eliminate inappropriate combination of settings using this technique.

Can I have multiple Excludes? Such as:

Exclude = ROCShortLookback < min1 OR ROCLongLookback < min2;
Exclude = Some Other Condition

It would depend if each Exclude acted as an immediate "return" during execution rather than "last setting" wins.

Thanks again!

We don't know and don't see your entire code.
You have to find out yourself via debugging when lookback becomes zero.

BTW, also take a look at Status() function.


Yes, you can make more lines

Exclude = ROCShortLookback < min1 OR ROCLongLookback < min2;
Exclude = Exclude AND Some Other Condition;

Or

Exclude = ROCShortLookback < min1 OR ROCLongLookback < min2;
Exclude = Exclude OR Some Other Condition;

Further details on this: I only get this error when I use the trib non-exhaustive optimization engine.

If I switch to cmae or spso, making no further changes to the code, the optimization runs to completion.

From previous comparison of the three engines I prefer the results from the trib engine, and have previously gotten a successful run using it. So something has changed, but I'm stumped as to what. I genuinely believe it is not a bug in my code.

I know the non-exhaustive optimization engines are third party code/dlls. Perhaps something has changed in my Amibroker or machine state? My last resort will be to reboot my computer and try again. If that doesn't fix the issue then I'm stumped and will use either cmae or spso.

Re: the last part of my original post: any further explanation of how the non-exhaustive optimization engines work may help me decide the best approach to optimizing a large optimization parameter set. I have done the usual "optimize in chunks", but I found optimizing the sum total of the optimization parameters together gave better "forward backtesting"/OOS backtesting results than the "chunked" approach, even though that approach was non-exhaustive.

I have calculated the dimensionality of my parameter set and the unique combinations are "huge". I am grateful for the non-exhaustive optimization engines, but even they will have their limits.

Or do they? If I configure:

OptimizerSetEngine("trib");  
OptimizerSetOption("Runs", 5); 			// default value 
OptimizerSetOption("MaxEval", 20000); 	// max number of evaluations  

Does that mean the engine runs 100000 tests (zeroing in on the best guess parameter combination), regardless of whether the optimization data set is 500,000 or 10 trillion? I assume the 1st one has the greater chance of being statistically accurate than the 2nd one.

Beyond my specific issue with my code, any general advice or best practice in optimizing a large optimization data set is most welcome.

In short: if these are your only Optimize() statements you should be using EXHAUSTIVE optimization. The number of combinations is way too small (only 528) for non-exhaustive. If not, the discussion is pointless until full formula is posted.

Hi Tomasz,

Thank you for your reply.

I'm unwilling to post my entire AFL on this forum, but a friend/commercial AFL developer reviewed my code. He only gave me a hint but it was enough to solve my problem. So thanks, you know who you are.

Here is the problem (this code is in a trailing stop loop):

        StopCond = 1;   
        for (j=0; j < StopCondDownBars; j++) {
            _ROC = ROC(Ref(Close,j),1);
            StopCond = StopCond AND _ROC[i] < 0;
        }

This should be:

        StopCond = 1;   
        for (j=0; j < StopCondDownBars; j++) {
            _ROC = ROC(Ref(Close,-j),1);
            StopCond = StopCond AND _ROC[i] < 0;
        }

i.e. negative j.

Here is my analysis and lessons learned for the future.

This is actually future looking: it is checking for the ROC on future bars instead of on previous bars. And if this happens near the end of the data, it will progress past the last bar, causing the error during optimization.

I click on the “AFL” icon (check syntax) thinking it does the code review. If I had clicked Analysis -> Code Check and Profile, it would have picked up this issue. A valuable lesson learned for the future: I should always run this on my code especially as it gets closer to production status.

Guessing here: The reason it appeared to be “fixed” with the spso or cmae engine is that all non-exhaustive engines use a random set of data, then “zero in” on the best results. Had I allowed these two engines to run longer they likely would have also encountered the problem.

I’m not sure why Amibroker said the error was in a completely different section of the code. But at the end of the day it was user error, both in the code and in my coding practice by failing to run "Code Check and Profile", esp. if my code is generating an error and doubly so before posting to these forums with a question.

On a more generic issue, I'm aware of the "Curse of Dimensionality". In fact, I calculated the dimensionality of all my optimization parameters in Excel. Easy to do, just enter the start, end, and step values, derive the range for that set, then use a formula to derive the product of the previous row with the current row.

The dimensionality is roughly 143 trillion.

I have been able to optimize "in chunks" using exhaustive optimization. However, I feel I get better results when I optimize the entire optimization parameter space together, even if the optimization was non-exhaustive. I also save the results into Excel, and it is easy to try different combinations of the optimized parameters and see which yield the best back test results.

Thanks again Tomasz for your reply. If you have further advice on best practice to optimize a large optimization data set, beyond what you've stated in https://www.amibroker.com/guide/h_optimization.html , please let me know. I'm especially interested in knowing more about how the non-exhaustive optimization engines work.

Kind Regards...

Thanks again Tomasz for your reply. If you have further advice on best practice to optimize a large optimization data set, beyond what you've stated in https://www.amibroker.com/guide/h_optimization.html , please let me know. I'm especially interested in knowing more about how the non-exhaustive optimization engines work.

This topic was automatically closed 100 days after the last reply. New replies are no longer allowed.