The SetCustomBacktestProc should work like this:
SetCustomBacktestProc( "Formulas\MyCustomBacktest.afl", True );
However, this causes two errors.
You have to use double backslashes or you will get error 54.
You have to include the entire file path, including the drive, or the backtester won't find the file.
The way to make it work with a filename is to include the entire filepath using double-backslashes:
SetCustomBacktestProc( "C:\Program Files\AmiBroker\Formulas\Backtests\CBT 20180912.afl", True );
This is how it works in my version of AB which is 6.30.
Is the Function Reference out of date? Or is this unexpected behavior of the function?
BTW because you have not used code tags/code button, your double backshlashes turned into single backshlashes. This is another example showing, that using code tags/code button on the forum is mandatory: