How is your process of IS-Paper Trade-OOS different than using AmiBroker's built-in walk-forward testing without an anchor date? It sounds to me like the parameters for each OOS backtest are completely determined by your "paper trading" results, and that the IS optimization before PT are not serving any real purpose.
Well, because the strategy must still perform well during the In-Sample period. Otherwise I could just optimize for the last 2 months instead of the full year. But, I want something that works well for the entire year, and then pick the one that is best on the final 2 months (or the following 2 months, if that is possible). I know it sounds rather pointless, but actually after using this feature for a looong time, it really works and helps dramatically with very complicated strategies to prevent them from over-fitting. My current strategy takes about 100,000 iterations to finish optimizing when using genetic optimization. If I manually plug in the settings and backtest on an OOS period, it is usually one of the "pretty good" optimizations in the middle of the range (like #20,000 or so) that will have the best OOS result. This is incredibly time consuming and tedious to change all the parameters and backtest manually, so having this done automatically during the optimization would be much better. And yes, I could also simplify my strategy and eliminate the need for this, but the more complicated version is much better OOS, even if it does overoptimize. It would be even better OOS if I had a better way to sort results using the "paper trading" feature.
Also, you don't need to compare EVERY single In-Sample optimization in the "Paper Trading" period - you could copy all the results to a spreadsheet and take the top 10% of the IS optimized results, then sort those by performance during the paper trade period. So you would be confident that the strategy would perform very well on the entire IS period, but then not be completely over-fit to the data since it would have to be the best result on the Paper Trading period.
And yes, the current walkforward functionality would do this same thing, but the way it works now is only the final "best" IS solution gets used for the OOS walkforward. I want to create an OOS walkforward for EVERY solution found in the In Sample period (or, the top 10%, whatever)
This thread looks like the solution:
Seems like this is close, but doesn't work:
function ProcessTrade( trade )
{
global tradedSymbols;
symbol = trade.Symbol;
//
if( ! StrFind( tradedSymbols, "," + symbol + "," ) )
{
tradedSymbols += symbol + ",";
}
//
// HINT: you may replace it with GetPercentProfit if you wish
//profit = trade.GetProfit();
profit = trade.GetPercentProfit();
//
if( trade.IsLong() )
{
varname = "long_" + symbol;
VarSet( varname, Nz( VarGet( varname ) ) + profit );
}
else
{
varname = "short_" + symbol;
VarSet( varname, Nz( VarGet( varname ) ) + profit );
}
}
//
SetCustomBacktestProc( "" );
//
/* Now custom-backtest procedure follows */
//
if ( Status( "action" ) == actionPortfolio )
{
bo = GetBacktesterObject();
//
bo.Backtest(); // run default backtest procedure
//
tradedSymbols = ",";
//Here is where I am trying to limit to a specific date window:
start = _DT( "2018-11-01" ); //Beginning of window
end = _DT( "2019-1-20" ); //end of window
//iterate through closed trades
for ( trade = bo.GetFirstTrade( ); trade; trade = bo.GetNextTrade( ) )
{
bi = BarIndex();
i = SelectedValue(bi) - bi[ 0 ];
_TRACE("current bar index "+i);
dt = DateTime();
datewindow = dt >= start AND dt <= end;
if (datewindow[i]) //only look at the date window
{
ProcessTrade( trade );
}
}
//
//iterate through open positions
for ( trade = bo.GetFirstOpenPos( ); trade; trade = bo.GetNextOpenPos( ) )
{
ProcessTrade( trade );
}
//
//iterate through the list of traded symbols and generate custom metrics
for ( i = 1; ( sym = StrExtract( tradedSymbols, i ) ) != ""; i++ )
{
longprofit = VarGet( "long_" + sym );
shortprofit = VarGet( "short_" + sym );
allprofit = Nz( longprofit ) + Nz( shortprofit );
// metric uses 2 decimal points and
// 3 (calculate sum) as a "combine method" for walk forward out-of-sample
bo.AddCustomMetric( "Profit for " + sym, allprofit, longprofit, shortprofit, 2, 3 );
}
}
Solved it 
Simple as expected, didn't know about "EntryDateTime" and "ExitDateTime". If anyone is interested in doing this, here you go -
function ProcessTrade( trade )
{
global tradedSymbols;
symbol = trade.Symbol;
//
if( ! StrFind( tradedSymbols, "," + symbol + "," ) )
{
tradedSymbols += symbol + ",";
}
//
//Here is where I am trying to limit to a specific date window:
start = _DT( "2018-08-01" ); //Beginning of window
end = _DT( "2019-1-20" ); //end of window
//dt = DateTime();
enterDate = trade.EntryDateTime;
datewindow = enterDate >= start AND enterDate <= end;
profit = 0;
if (datewindow) //only look at the profit from trades which are entered during the datewindow
{
// HINT: you may replace it with GetPercentProfit if you wish
//profit = trade.GetProfit();
profit = trade.GetPercentProfit();
}
if( trade.IsLong() )
{
varname = "long_" + symbol;
VarSet( varname, Nz( VarGet( varname ) ) + profit );
}
else
{
varname = "short_" + symbol;
VarSet( varname, Nz( VarGet( varname ) ) + profit );
}
}
//
SetCustomBacktestProc( "" );
//
/* Now custom-backtest procedure follows */
//
if ( Status( "action" ) == actionPortfolio )
{
bo = GetBacktesterObject();
//
bo.Backtest(); // run default backtest procedure
//
tradedSymbols = ",";
//iterate through closed trades
for ( trade = bo.GetFirstTrade( ); trade; trade = bo.GetNextTrade( ) )
{
ProcessTrade( trade );
}
//
//iterate through open positions
for ( trade = bo.GetFirstOpenPos( ); trade; trade = bo.GetNextOpenPos( ) )
{
ProcessTrade( trade );
}
//
//iterate through the list of traded symbols and generate custom metrics
for ( i = 1; ( sym = StrExtract( tradedSymbols, i ) ) != ""; i++ )
{
longprofit = VarGet( "long_" + sym );
shortprofit = VarGet( "short_" + sym );
allprofit = Nz( longprofit ) + Nz( shortprofit );
// metric uses 2 decimal points and
// 3 (calculate sum) as a "combine method" for walk forward out-of-sample
bo.AddCustomMetric( "Profit for " + sym, allprofit, longprofit, shortprofit, 2, 3 );
}
}
I am still interested in finding out if there is a way to do this for dates outside of the optimization range I'm using. If anyone has an answer to that I'd love to know! Thanks
2 Likes
And a much simpler way to do this, but only gives total portfolio profit:
SetCustomBacktestProc( "" );
if ( Status( "action" ) == actionPortfolio)
{
//find profit from start date to end of backtest period
start = _DT( "2018-01-01" ); //Beginning of window
bo = GetBacktesterObject();
bo.Backtest(); // run default backtest procedure
start_eq = Lookup(bo.EquityArray,start);
end_eq = bo.equity;
total_profit = 100*(end_eq-start_eq)/start_eq;
bo.AddCustomMetric( "Date Window % Profit", total_profit, 2);
}
1 Like
And, even better... instead of setting the beginning of the window, now you just set the window length in days, and this will calculate the % profit for the last X days of your optimization. This can then be used as the optimation target in walkfoward tests. Lets see how much better walkforward results this gives than using the CAR/MDD of the entire period 
I think best would be to multiply the "window profit' by CAR/MDD, and use that as optimization target. And I'll call it "CAR Window"
// Window Profit: Gives % profit for last "windowlength" days of an optimization.
SetCustomBacktestProc( "" );
if ( Status( "action" ) == actionPortfolio)
{
//find profit for window in backtest
windowlength = 60; //window length in days
//start = _DT( "2018-10-01" ); //Beginning of window
range_end = Status( "rangetodate"); //gives end of range in datenums
range_end = DateTimeConvert( 2, range_end); //converts DateNum format to DateTime format
start = DateTimeAdd(range_end,-windowlength);
bo = GetBacktesterObject();
bo.Backtest(); // run default backtest procedure
start_eq = Lookup(bo.EquityArray,start);
end_eq = bo.equity;
window_profit = 100*(end_eq-start_eq)/start_eq;
st = bo.GetPerformanceStats(0); // get stats for all trades
carmdd = st.GetValue( "CAR/MDD");
carwindow = carmdd*window_profit;
bo.AddCustomMetric( "Window % Profit", window_profit, 2);
bo.AddCustomMetric( "CAR Window", carwindow, 2);
bo.AddCustomMetric( "Window Length (Days)", windowlength);
}
Well, actually this still doesn't do what I need to do, since I'm using the result from my "Paper trading" period as the optimization target, so that defeats the purpose of this whole thing. I'll try to give a better explanation of this thing.
Paper trading has 3 windows of data. In-Sample, "Paper trading" , and Out of Sample. The optimization engine should only use fitness metrics from the "In Sample" period as the optimization target. THEN - these results are sorted using metrics calculated from the "paper trading" period - but only after the optimization is complete. THEN the selected result is used to run a backtest on the Out Of Sample period. So, it's 3 seperate steps, the "Paper Trading" period is not included as part of the optimization target.
I can see how it would be possible to manually do this by splitting the date range into 2 "windows", using metrics calculated from the first "window" as the optimization target, then manually sorting the results by metrics calculated from the second window. Is there any way to automate this though?
If anyone can help with this, I really appreciate it. If I am remembering correctly, it really does give more reliable OOS performance to do this 2-step sorting process. Being able to verify this using the built-in walkforward testing of amibroker would be wonderful. Thank you very much
I think my post above gives a better explanation of this concept. The Paper Trading result is not used by the optimization engine, it's only used to sort the optimized results AFTER optimization is complete as a second "verification" step before running the OOS backtest for the walkforward.
Each part of the test uses a seperate section of data. Here is a visual: 
Does not make sense to me. What's the gain from this "paper trading" curve? It is against walk-forward principles where you have in-sample and out-of-sample period coming right after in-sample.
These paper trading periods do NOT add anything meaningful.
But if you really want to do that you can do so with existing functionality - in the Walk Forward settings page define custom IIS and OOS period so there are holes between IIS and OOS. These holes will be your "paper trading" that don't influence anything.
1 Like
Great thank you! Now how can I sort the optimizations by how well they perform during the Paper trading period before backtesting on the OOS period?
Again, the paper trading period is a "filter" for your optimization results. You pick the one that performs best on the Paper trading period, then use that set of parameters to backtest on the OOS period. You don't just take the best result from the In-Sample period, then use that on the OOS period. Do you see what I'm saying?
Example:
1 year long In-Sample, 2 week "Paper Trade", then 2 month "OOS" for my walkforward test.
set CAR/MDD (for the In-Sample period) as the optimization target, optimize the strategy. For each backtest that runs during the optimization, we also calculate the built-in backtestobject metrics for the "Paper Trade" period and display those results on the optimization screen too, in seperate columns.
Once the optimization engine is satisfied that it has the best CAR/MDD solution for the In-Sample period, but BEFORE running the OOS backtest, we sort all of those optimizations by the % profit (or whatever metric you want) during the paper trade period. THEN run the OOS backtest using THAT set of parameters, NOT the best "CAR/MDD" from the In Sample data.
There is no built in way to do this. Please read and understand what I just wrote before replying. Thank you for the help.
Suppose I have a strategy that I know is incredible and amazing but is prone to high drawdowns if you don't get the settings right, and I'm running a fresh optimization to update my live trading instance. So, I run an optimization on the past 10 months of data up until a week ago. Then, just to double check that these are good settings and will work well in the current market, I run a week-long backtest on the most recent market data, using the parameters from the best optimization result. IT LOSES 50% IN A WEEK!!! Do I want to run my strategy live with these settings??? HELL NO.
I'd like to take the top 50% of my optimization results from the In-Sample period and sort them by how well they would have performed during the past week. If I have a good optimization result that performs pretty well on the past week, I'd choose that set of parameters to run live, instead of an INCREDIBLE optimization result that loses 50% in the past week. Wouldn't you?
That is the point of this "paper trading" period in the walkforward test. Final verification/sorting of the In Sample optimization results before running the strategy live (or backtesting on the OOS period for a walkforward test)
Right now, I just do this manually, and very tediously, by typing in the parameters and backtesting my strategy for the most recent data, before running them live. If it majorly screws the pooch in the past week or 2, I don't use those results. This process takes forever and is impossible to do this for a walkforward test unless I want to do it manually for each step of the walkforward test, which would take many multiples of forever.
Can anyone help?
Thank you again for your time, really appreciate the assistance.
+1
It would make more sense to perform a 'Robust Backtest' over a deep IS data period (multiple years covering all different market conditions) and then testing those rules on a non-contaminated OOS data period) rather than just doing a basic non-robust 'Backtest'.
There is really no need for the "Paper Trading" section between the IS and OOS data periods. But each to their own...
Well, if your strategy works best by optimizing on years and years and years of data, that's great. Mine doesn't. But, to each his own.
The way my strategy works is it gets optimized for 6 months to 1 year of data, then run live. It doesn't work optimizing it on multiple years, because the market changes too fast.
If I can filter those optimization results by backtesting on a week or 2 of data before running live, the results increase dramatically. Whether you understand why this works, or not, or whether you believe me, or not, doesn't really matter. It works, and it improves live-market results dramatically. I'm tired of doing this manually, it takes for ever and it's a PITA.
My question is, is it possible to automate this in the .AFL code, or not? I was told the forum is where you get help with writing code since support doesn't do that anymore. Well, I need some help writing code, I'm not a good coder. Is anyone able to help with my problem?
Any HELP solving this problem would be hugely and greatly and deeply appreciated. Heck, you might even give it a try yourself and see how amazingly much of an improvement this makes to walkforward tests, if we can get something that works. IT MAKES A HUGE DIFFERENCE, whether you believe me or not.
Thank you!
Also, my "robust backtest" is called a Walkforward Test. I perform a Walkforward Test (https://www.amibroker.com/guide/h_walkforward.html) over multiple years, which gives me the total OOS profit during that time, and then I can see how my strategy performs in real life. Then I do that 100 more times. I adjust the In Sample optimization lengths, and the Out of Sample lengths, and find that I get the best OOS results with my strategy by optimizing on about 6-12 months of data, and re-optimizing the settings once every week - month.
I also found that using the "PAPER TRADING" feature that the "other trading software" has makes my OOS results increase BY A FACTOR OF 10 or more. So, I would love to do that in Amibroker too, since on all other counts aside from the lack of this one single feature, it is VASTLY superior. But, as far as I can tell, there is no way to make it work other than writing some DLL that I have no ability to code. Pretty frustrating. Hopefully someone will actually try to help solve this problem instead of just repeating over and over how they don't understand how this feature that they've never used and that they don't really understand what it does isn't going to be useful. IT IS USEFUL.
If anyone is capabable and willing to actually help solve this problem instead of just repeating how you don't understand why or how it can be useful, that would be lovely. Help solving this would be greatly apprecialted.
.
.
.
.
Or, alternatively, if anyone can point me in the direction of a great AFL coder who I could just hire to help me with this problem, that would be great too. Would prefer to figure it out in the forum here and have a solution here that everyone can use, but I'm willing to just hire someone to code it for me if that's not an option.
I'm familiar with how this feature should work, and I'm also quite familiar with how effective it is. So, please stop telling me how this feature that I've used with tremendous success for many years and you've never used before is not beneficial. It's not helpful.
It's really quite simple and I'm not sure why people seem to be having such a difficult time grasping the concept. You optimize your strategy with a certain fitness metric. Then you filter those results based on how well they perform in non-optimized data. Then you run it live. It's a 3-step process instead of a 2-step process you are familiar with in normal walkforward testing.
Here's a quick refresher on walkforward testing: https://www.amibroker.com/guide/h_walkforward.html
Instead of the 1-2, 1-2, 1-2 process shown in the animation, it's a 1-2-3, 1-2-3, 1-2-3 process. Optimize, verify/select best optimization, then OOS backtest. Simple. Easy.
At least, it should be. I'd love to see how this can be done with Amibroker. Please show me.
Thanks
Until a solution of how to create a 3-step Walkforward test is found, I've been using my "Window Profit" custom metric code to definite a better optimization target. Same code I posted previously but edited slightly. I define a window of X days, which gets subtracted from the last day of the from-to date range specified in the optimization or In-Sample walkforward step to define my "Date Window". To keep things simple/stupid, I just lookup the equity at the beginning of the date window and the end of the range of dates used in the optimization, and calculate profit, and (end/start)%
//Custom Metrics
// Window Profit: Gives % profit for last "windowlength" days of an optimization.
SetCustomBacktestProc( "" );
if ( Status( "action" ) == actionPortfolio)
{
//find profit for a date window in backtest
windowlength = 60; //date window length in days
//start = _DT( "2018-10-01" ); //Beginning of window
range_end = Status( "rangetodate"); //gives end of range in datenums
range_end = DateTimeConvert( 2, range_end); //converts DateNum format to DateTime format
start = DateTimeAdd(range_end,-windowlength); //subtract windowlength from end date in DateTime format
bo = GetBacktesterObject();
bo.Backtest(); // run default backtest procedure
st = bo.GetPerformanceStats(0); // get stats for all trades
start_eq = Lookup(bo.EquityArray,start); // lookup equity value at start of user-defined date window
end_eq = bo.equity; // get equity at range end of backtest
window_change = Nz(100*end_eq/start_eq); //ending equity divided by starting equity % for date window
window_profit = Nz(100*(end_eq-start_eq)/start_eq); //profit % calculated for date window
carmdd = Nz(st.GetValue( "CAR/MDD")); //CAR/MDD for entire date range of the backtest
carchange = IIf(carmdd>0,carmdd*window_change,0);
carchange2 = carchange*window_change;
carwindow = IIf(carmdd>0 AND window_profit>0,carmdd*window_profit,0);
carwindow2 = carwindow*window_profit;
bo.AddCustomMetric( "Window % Profit", window_profit, 2);
bo.AddCustomMetric( "CAR/MDD * Window Profit", carwindow, 2); //previously "CAR Window"
bo.AddCustomMetric( "CAR/MDD * Window Profit^2", carwindow2, 2); //previously "CAR Window2"
bo.AddCustomMetric( "CAR/MDD * Window Change ", carchange, 2); //previously "CAR Gain"
bo.AddCustomMetric( "CAR/MDD * Window Change^2", carchange2, 2); //previously "CAR Gain2"
bo.AddCustomMetric( "Window Length (Days)", windowlength);
}
Currently I'm using "CAR/MDD * Window Change^2" as my optimization target (called "CAR Gain2" in the image below), and comparing that to just using CAR/MDD as the optimization target.
The walkforward test is only a few steps in (It takes a few days to finish) but so far the results look positive. A dramatic increase in the OOS results for the first month compared to the CAR/MDD optimization target, and only a slight decrease for the second month.
CAR Gain2 (CAR/MDD * Window Change ^2) optimization target:
CAR/MDD optimization target:
Now, if I could implement the full 3-step procedure I'm trying to do in amibroker, and use CAR/MDD as the optimization target (step 1), then SORT those results by their performance on a short "Paper Trade" OOS period (step 2), THEN backtest the best of those on the OOS period (step 3), the OOS results would go up by a factor of 2 - 10. I know this from experience.
If someone more skilled with coding can help solve this problem or point me in the direction of someone who I can pay for help, please do and thank you very much! I'm guessing it's going to take a custom dll to implement this walkforward procedure since amibroker uses a 2-step walkforward and there doesn't appear to be any way to calculate metrics based on an Out-of-Sample date range for each optimization step, and use a different metric than the optimization target to select between optimization results. Or maybe I'm mistaken and there is a way to do it without a .dll (Hope so!!)
Take care guys!
1 Like
I think that you could achieve your goal through clever use of a couple of static variables. For purposes of a conceptual discussion, let's not worry about the most efficient implementation for now, and let's assume you only want 10 strategy variations to survive the paper trading screening process.
Assume we have three static array variables, of which we will only use 10 positions. A 10x3 matrix would also work.
- IS Score: The value of your ranking metric for the IS period without the paper trading period at the end.
- PT Score: The value of (a perhaps different) ranking metric for the paper trading period.
- Best Opt Target Values: The optimization target value reported to AB for the current 10 best strategy variations. This is a value that we will make up as we go. It has no meaning/use other than to score relative "goodness" of a variation.
Initialize all of these to zero at the start of each new walk-forward step, i.e. the start of each new iteration of the full optimization.
If this is the first variation in the optimization, store the IS Score and PT Score in the static arrays. Set the Opt Target to 1000.
For variations 2-10, store the IS Score and PT Score in the static arrays. Set the Opt Target to a value that reflects the correct position of the PT metric relative to the other variations that have already been run. For example, if variation 2 has a PT metric that is less than variation 1, then give it an Opt Target value of 900. If variation 3 ends up between variations 1 and 2, give it an Opt Target value of 950, etc. Store these values in the Best Opt Target array/matrix.
For variations 11-N, generate the IS and PT scores/metrics. If the IS score is better than 1 or more of the existing Top 10 IS scores, then the current variation should replace the worst variation on the Top 10 list. Update all three arrays to reflect this replacement operation, and assign an appropriate Opt Target value to show how the PT score for this variation ranks against the other PT scores.
If the IS score is not better than any of the existing Top 10 IS scores, then simply assign an Opt Target value of 0, because we don't want AB to select this variation anyway (we did not pass the first screening).
All three of these cases (variation 1, variations 2-10, Variations 11-N) could really be collapsed int a single case, but I called them out here for clarity.
When you're done, the IS Score array will hold the metrics for 10 variations that had the best performance during the IS period. The PT Score array will hold the paper trading metrics for those same variations. And the Best Opt Target Values will hold the 10 largest values that you reported to AmiBroker as your optimization targets.
Created a new thread here which has a more appropriate title and is more to the point of what I'm trying to achieve in the walkforward testing -
Also, my "robust backtest" is called a Walkforward Test. I perform a Walkforward Test (https://www.amibroker.com/guide/h_walkforward.html) over multiple years, which gives me the total OOS profit during that time, and then I can see how my strategy performs in real life. Then I do that 100 more times. I adjust the In Sample optimization lengths, and the Out of Sample lengths, and find that I get the best OOS results with my strategy by optimizing on about 6-12 months of data, and re-optimizing the settings once every week - month.
I also found that using the "PAPER TRADING" feature that the "other trading software" has makes my OOS results increase BY A FACTOR OF 10 or more. So, I would love to do that in Amibroker too, since on all other counts aside from the lack of this one single feature, it is VASTLY superior. But, as far as I can tell, there is no way to make it work other than writing some DLL that I have no ability to code. Pretty frustrating. Hopefully someone will actually try to help solve this problem instead of just repeating over and over how they don't understand how this feature that they've never used and that they don't really understand what it does isn't going to be useful. IT IS USEFUL.
If anyone is capabable and willing to actually help solve this problem instead of just repeating how you don't understand why or how it can be useful, that would be lovely. Help solving this would be greatly apprecialted.
.
.
.
.
Or, alternatively, if anyone can point me in the direction of a great AFL coder who I could just hire to help me with this problem, that would be great too. Would prefer to figure it out in the forum here and have a solution here that everyone can use, but I'm willing to just hire someone to code it for me if that's not an option.
I'm familiar with how this feature should work, and I'm also quite familiar with how effective it is. So, please stop telling me how this feature that I've used with tremendous success for many years and you've never used before is not beneficial. It's not helpful.
It's really quite simple and I'm not sure why people seem to be having such a difficult time grasping the concept. You optimize your strategy with a certain fitness metric. Then you filter those results based on how well they perform in non-optimized data. Then you run it live. It's a 3-step process instead of a 2-step process you are familiar with in normal walkforward testing.
Here's a quick refresher on walkforward testing: https://www.amibroker.com/guide/h_walkforward.html
Instead of the 1-2, 1-2, 1-2 process shown in the animation, it's a 1-2-3, 1-2-3, 1-2-3 process. Optimize with optimiztion target fitness metric during IS period. Verify and select best parameter set with a DIFFERENT metric during "PaperTrade" period, then perform the OOS backtest. Simple. Easy.
At least, it should be. I'd love to see how this can be done with Amibroker. Please show me.
Thanks