Issues with Custom CBT for Walk Forward Stats (using static variables)

Hi all, I'm attempting to reference IS walk forward monte carlo stats (from best IS optimization step) to pass to the OOS AFL for position sizing.

Specifically, let's say I want to use MC 1 percentile MDD of the best/chosen IS optimization for the AFL to determine position sizing in the following OOS step. The reason I'm trying to do this is because it's what I'd like to do in live trading (using MC stats of the last IS period as a factor for position sizing).

I've had limited luck with passing data from CBT to AFL through static variables, however I've run into numerous challenges while doing so.

Challenges include:

  1. Not sure how to tell if backtester is on IS or OOS step to designate for static vars
  2. OOS AFL formula gets the last optimization value from IS, not the best/chosen
  3. Not sure how to tell what step the walk forward is on (first or last especially) to destroy/remove static vars as desired

I've read all the CBT documentation I could find on the site and forum, however that's not to say that I haven't missed some important details or that what I want to do is even possible currently.

Here's the premise of my code. I'm also passing the value AFL uses back to CBT so it can be displayed in WF results.

//CBT
#pragma enable_static_decl(cbt)
SetCustomBacktestProc("");
if( Status("action") == actionPortfolio )
{
...
    mc = bo.GetMonteCarloSim();
    if( mc )
	{
		static _MDD1;
		_MDD1 = mc.GetValue( "MaxSystemDrawdownPercent", 1 );
		bo.AddCustomMetric( "MDD1", _MDD1);
		bo.AddCustomMetric( "PrevMDD1", StaticVarGet("afl_MDD1"));
	}
}

//AFL
#pragma enable_static_decl(afl)
...
static _MDD1;
_MDD1 = Nz(StaticVarGet("cbt_MDD1")); 
...position sizing etc. based on MC stat

Sort of a workaround I've found has been to use a "dummy" optimization parameter and no real optimization params that affect the result. That way I get the best (only) results in walk forward. However, in the IS optimization steps, they're using previous step static vars instead of desired (or none).

Ideally, I'd like to destroy/remove the static vars for IS optimization steps and only save the best IS MC stats for use in OOS AFL formula, if that makes sense. I may also want to use the previous OOS MC stats for following IS AFL position sizing, but that's secondary.

Has anyone had luck with using walk forward IS MC stats in their AFL formulas or would you have advice on the above?

Thanks for reading.

A couple of points to clarify. By best/chosen, I mean the IS optimization chosen during walk forward based on optimization target. Also on #2 I mean to say the MC stats I'm looking to get.

The software is amazing, by the way. It runs incredibly fast and does much more than I know how to do with it.

Figured a possible workaround referencing "~~~OSEQUITY" and "~~~ISEQUITY" but it may not be efficient for me to do it this way. Essentially in CBT, if current final equity == "~~~OSEQUITY" then the backtester is on OOS step. The harder part is letting the AFL know that it's on OOS step since it's getting the previous values static var stats and there are multiple IS steps normally. Basically if I set a step/sample/stage static var that says the prev step was IS then the current for AFL could either be another IS or the OOS. It also doesn't tell me which IS step was chosen.

This may be hard to follow without an example, but I could possibly use an array or matrix to hold IS equity, [optimization target], and any MC values from CBT and then in the AFL match up the IS equity part of the array/matrix to "~~~ISEQUITY" (since it should be updated with optimal last value if this is OOS) and use the corresponding MC stats in the AFL.

Basically, if not OOS (static var from CBT) AND there's a match for IS equity array/matrix static var, then I say the AFL is in OOS processing. Then it does position sizing based on MC stats. Then destroy the IS equity array/matrix static var.

Hopefully there's a better way, but it'll be a neat coding exercise either way. Have done some testing along these lines already. Thanks again. Btw, I looked to edit the post above instead of following up, but it didn't appear to be an option. So I apologize for replying already without allowing time for others to consider.

A roadblock I've been hitting is that "~~~ISEQUITY" and "~~~OSEQUITY" don't appear to be updating for me. Tested using LASTVALUE or ENDVALUE and also by looking at the chart. They were last updated in March and 4/1 despite my daily testing and scripting. Checked the app settings and also tried my other machine, but it was a similar thing with them not updating like I would expect during or after WF testing. Then again, I may be missing something about them as well.

If I can figure a way to tell first WF step and which AFL runs are in OOS (from within AFL), that would definitely be a massive help. That way I could run optimization parameters and also use the MC stats as variables during walk forward. Alternatively, if I could reference the first step and how many optimization runs are in each IS step, that would also allow me to achieve the same. The other challenge is the MC values of course, but I think a matrix can be used and sorted to find the "chosen" IS run values.

There's more or less rigging my approach by restarting Amibroker to reset static vars and using a counter for the walk forward tests, but it's not ideal for obvious reasons and others. It becomes a burden, especially when setting up a matrix as well. Still a world better than doing separate MC analysis and integrating, though.

Usually I'm good at pushing boundaries, it's just that I'm not that good in this case. :sweat_smile:

Please follow this advice: How to ask a good question and provide

minimum, but working reproducible example of the formula that shows the problem that can be copy-pasted and run without modification

One thing that you should be aware is that you can NOT have more than one #pragma enable_static_decl per formula.

You are using it twice with different prefixes and that is wrong.
#pragma is pre-processor command and it is NOT executed within normal program flow.
There must be ONE AND ONLY ONE #pragma enable_static_decl at the top of the formula and nowhere else. And only one auto-added prefix per formula is allowed.

1 Like

Are your IS and OOS backtests the same length? If not, it seems that you could BarCount in the CBT to tell whether you're currently running IS or OOS.

Way back in 2009, I had a similar problem and implemented a function called OptSampleType() that returns the in-sample / out-of-sample stage and some range info. It modelled those stages with a simple state machine. I called it a "hack" at the time because it is dependent on the internal steps and the order that Amibroker takes. I was hoping long term that Tomasz would implement such a function, but it has worked for all those years, though.

I'm away from my home office, but if you drop me an email, I'll dig it out.

FWIW, it was part of another effort to capture "best" optimization info, including parameters, in the CBT during the optimization. If you go down this path, be prepared to write some non-trivial code.

This topic was automatically closed 100 days after the last reply. New replies are no longer allowed.