Walk forward optimization with custom criteria

I red this,
https://www.amibroker.com/guide/a_custombacktest.html

I am optimizing a portfolio with WFO and a custom metric.
It is not clear to me if it possible to add a custom metric like for example the CAR of the last year when for example having an in sample period of 10 years. It is usefull to know if the strategy also worked in sample at the end of the in sample period. I see that option in other software too and it
can filter out systems that did work in the past but currently do not work anymore.
From the documentation it seems that I can only add the inbuild custom metrics but I hope I am wrong. Thanks for your help or directing me how I could do it.

metric

2 Likes

I use a parameter here that is a custom fit criteria like below.

Test = NP * PF^1 * PayoffRatio^0.5 * AllQty^0.5 * (abs(CARdivMDD))^1

Apart from that I specify for example AllQty and CARdivMDD
to be a minimum value otherwise I set Test tot zero.
In this way only more valid settings are walked forwarded.
When optimizing a parameter setting I often find that OOS is better when
certain minimal performance parameters are reached. One of them is the performance of the last year when for example IS is 10 years.
Another is often the number of trades and profit factor, winning% trades.

I find that optimizing only on net profit can come up with not
significant results like very long trades and few trades. It all
depends on the system being tested. There must be enough
trades to have some significance.

I usually use the Tribes optimizer and run between 5000-20000 parameter settings depending on the number of parameters and length of IS.

I am searching for a way to have a new custom parameters something like the IS performance of the last 5-10% of the IS backtest.
You could then filter the settings out that do not work anymore the
last few months, year etc.
Would it be possible to add such a custom criteria parameter that does
only work on a part of the IS data?

A custom criteria setting and filter on it that show good in 10-20 walkforwarded settings is more likely to keep up when using that system live.

Always when optimizing you have the chance of getting parameter setting based on luck or setting that do currently not work anymore. That is just a disadvantage when using a smart the optimizer.
There is one tric used by a program that I use of which I was very sceptical in the beginning but it works.

It is dividing the IS data in 2 parts, the 1ste part that is used for parametric optimization and a second unseen IS part that is used for performance check IS on unseen IS data. It sounds weird but it works. So for example when having 10 years of data you could divide the IS data in for example even monts/weeks and uneven weeks. When daytrading you could for example set it to every second day. Parameter optimization is done in the first "real" IS period and confirmation is done in the second unseen IS period. Only if confirmation has a similar performance it is walk forwarded. By walkforwarding many times and looking at parameter stability generated systems can be selected as a group. After that you can further check that group of systems on other markets and filter only the ones that work robust. I can't mention the name of that program on this NG but it is definitely more than convincing enough to try if I could do similar things in Amibroker.