Unattended rolling X year backtests

Is there a way to get Amibroker to do unattended rolling X year backtests? This is not walk forward.

For example:

  • 2000-01-01 - 2003-01-01
  • 2001-01-01 - 2004-01-01
  • 2002-01-01 - 2005-01-01
    ...
  • 2018-01-01 - 2021-01-01

Go to bed and review the backtest results in Report Explorer in the morning?

Perhaps via the Amibroker Object Model and OLE scripting?

@LinusVanPelt, the short answer is yes.

Search the forum for "batch xml," and you'll find some threads that consider that AmiBroker projects are XML files.

This means that you can write an external batch procedure (using .js, python, or any other scripting supported language) to loop over an array of dates and, at each iteration, modify the .apx file accordingly before launching it via OLE.

As an alternative to using OLE, in your script, you can also invoke via command line a simple AmiBroker batch (.abb) that will load the project and will run the backtest.

In particular, I think that this past thread may be helpful to you.

This can also be done without batch, very easily, using Walk forward optimizer by using Advanced mode and turning off "In-sample" data:

image

Update: Sorry, I posted this without checking, but message about missing opt parameters from missing IIS step blocks this unusual way of using WF optimizer.

8 Likes

Thanks @beppe and @Tomasz. Much appreciated.

Edit: Do I have to specify an optimization parameter? I don't want to re-optimize after each run, I just want to move the date range forward.

If I have to specify an optimization parameter, what's the fastest approach?

Something like?

WFDummy = Optimize("WFDummy",0,0,1,1);

But then doesn't this double the time of execution, once for each setting?

The code I'm testing this on has these lines:

ROCLongPeriod           = Optimize("ROCLongPeriod", 120, 90, 150, 10); 
ROCLongCutoff           = Optimize("ROCLongCutoff", 15, 12, 18, 1); 

but I'm getting this error:

image

Sorry apparently if you only allow out-of-sample periods it looks for optimum parameters (from missing In-sample step) and finds none hence the error message that blocks from continuing. So it won't work the way I thought (maybe in the future version - should be just a matter of NOT displaying an error in this special case).

Hi All,

I got this to work, at least how I wanted. I thought I'd share my Powershell script with the community.

See the program header for full documentation.

If there's a better approach or you find bugs please let me know.

<# HEADER
/*=====================================================================
Program Name            : RollingBacktests.ps1
Purpose                 : Run Rolling Backtests in Amibroker using a
                          specified StartYear, EndYear, and StepYear.
Powershell Version:     : 5.1.19041.906
Input Data              : N/A
Output Data             : N/A

Originally Written by   : Linus Van Pelt
Date                    : 05MAY2021
Program Version #       : 1.0

=======================================================================

Modification History    : Original Version

=====================================================================*/
#>

<#
.SYNOPSIS
Rolling Backtests

.DESCRIPTION
Run Rolling Backtests in Amibroker based on StartYear, EndYear, and YearStep

.PARAMETER  StartYear
The starting year (From-Date) for the series of backtests

.PARAMETER  EndYear
The ending year (From-Date) for the series of backtests

.PARAMETER  StepYear
The step size (in years) for the series of backtests

.PARAMETER  MonthDay
The Month and Day for the rolling backtests, specified as "MM-DD" (eg. 12-25)

.PARAMETER  InputAPX
The Input APX file (Analysis File) to be backtested

.PARAMETER  OutputAPX
A temporary APX file with modified From-Date and To-Date, used to implement the rolling backtests

.PARAMETER  OutputCSV
A filename template used for the (optional) output CSV file.
The filename template should be something like "Rolling Backtest - {0} - {1} to {2}.csv"
where {0} = Formula filename, {1} = StartYear, and {2} = EndYear.

.PARAMETER  NoCSV
A switch to turn off the generation of the output CSV file.
The OutputCSV filename template is ignored when NoCSV is specified.

.PARAMETER  Quiet
A switch to turn off additional messaging to the terminal and run in "quiet" mode.

.PARAMETER  Whatif
Displays what would happen if you executed the script, without actually executing the script.

.PARAMETER  Confirm
Asks for confirmation before actually executing the script

.PARAMETER  Verbose
Prints additional diagnostic information to the terminal.

.EXAMPLE
.\RollingBacktests.ps1

Description
-----------
Runs the Rolling Backtests with the program defaults.

.EXAMPLE
.\RollingBacktests.ps1 2000 2020 3
.\RollingBacktests.ps1 -StartYear 2000 -EndYear 2020 -StepYear 3

Description
-----------
Runs the Rolling Backtests starting with From-Date of 2000-01-01,
ending with From-Date of 2020-01-01, with a Step Size of 3 years,
using the remaining program defaults.

.EXAMPLE
.\RollingBacktests.ps1 2000 2020 3 "12-25"
.\RollingBacktests.ps1 -StartYear 2000 -EndYear 2020 -StepYear 3 -MonthDay "12-25"

Description
-----------
Runs the Rolling Backtests starting with From-Date of 2000-12-25,
ending with From-Date of 2020-12-251, with a Step Size of 3 years,
with the Month and Day anchored on 25Dec,
using the remaining program defaults.

.EXAMPLE
.\RollingBacktests.ps1 -InputAPX .\MySystem.apx -OutputAPX .\RollingBacktests.apx

Description
-----------
Runs the Rolling Backtests using the specified Input APX file,
writing the modified APX file to the specified Output APX file,
executing the Rolling Backtests using the specified Output APX file,
using the remaining program defaults.

.EXAMPLE
.\RollingBacktests.ps1 -OutputCSV ".\{0}-MyWatchlistName-{1} to {2}.csv"

Description
-----------
Runs the Rolling Backtests using the specified filename template
for the output CSV file.
If you do not specify the {0}, {1}, and {2} placeholders,
you will need to make minor modifications to the script.

.EXAMPLE
.\RollingBacktests.ps1 -NoCSV

Description
-----------
Runs the Rolling Backtests using the program defaults,
but does not generate the Output CSV trades list.
The backtest results can be viewed using Report Explorer.

.EXAMPLE
.\RollingBacktests.ps1 -Quiet

Description
-----------
Runs the Rolling Backtests using the program defaults in "quiet" mode.

.EXAMPLE
.\RollingBacktests.ps1 -whatif

Description
-----------
Shows what the program would do, without actually executing the Rolling Backtests.

.EXAMPLE
.\RollingBacktests.ps1 -confirm

Description
-----------
Requires you to confirm each Rolling Backtest.
This is not usually what you want, but would allow you to skip certain backtests.

.EXAMPLE
.\RollingBacktests.ps1 -verbose

Description
-----------
Runs the Rolling Backtests with additional diagnostic information in the terminal.

.NOTES
This program runs a backtest from the StartYear to the EndYear,
with the specfied StepYear for the number of years.

For example, StartYear=2000, EndYear=2002, StepYear=3 would run backtests from:

   * From-Date: 2000-01-01 To-Date: 2003-01-01 (so data from 2000, 2001, 2002)
   * From-Date: 2001-01-01 To-Date: 2004-01-01 (so data from 2001, 2002, 2003)
   * From-Date: 2002-01-01 To-Date: 2005-01-01 (so data from 2002, 2003, 2004)

So remember, StartYear and EndYear sets the From-Date setting only,
and the StartYear + StepYear sets the To-Date setting.

I have only designed this script for yearly increments, with the month and day
anchored to the MonthDay parameter setting.

You can name the OutputAPX anything - Amibroker names the backtest results based on the formula name.

In most cases, you should keep the general format for the OutputCSV filename template.
If you do not increment the years in the output filename, you would likely overwrite the 
output from previous invocations.  If you don't like the default filename prefix the
easiest thing to do is edit the script and change the default value to your liking.
However, it could be useful to include the watchlist or filter criteria in the output CSV filename
if you wanted to backtest over multiple watchlists.

In fact, you could save say three versions of your InputAPX file:

   * My System - All Ordinaries.apx
   * My System - S&P ASX 200.apx
   * My System - S&P ASX Small Ordinaries.apx

Then invoke the script as:

"All Ordinaries","S&P ASX 200","S&P ASX Small Ordinaries" | `
   % {.\RollingBacktests.ps1 2000 2020 3 -InputAPX ".\MySystem - $_.apx" -OutputCSV "MySystem - $_ - {0} - {1} to {2}.csv"}

$_ is replaced with each token in the pipeline, and the script will be invoked for each saved APX file.

I have not invested the effort to make this script completely bulletproof by trapping for all possible errors.
If you input bad parameters, such as 1995.5 or ABCD for StartYear, or "25-12" for MonthDay,
the script may or may not trap the error.

If you find the script is not doing what you wanted, first end the script by pressing Cntl-C in the terminal.
Then review the open windows in Amibroker.  You likely would want to manually close the OutputAPX window
before re-running the script.
#>

#region Parameters
[CmdletBinding(SupportsShouldProcess=$true)]
Param(
   [ValidateRange(1990,2030)]
   [int]$StartYear=2000
   ,
   [ValidateRange(1990,2030)]
   [int]$EndYear=2022
   ,
   [ValidateRange(1,40)]
   [int]$StepYear=3
   ,
   [ValidateNotNullOrEmpty()]
   [string]$MonthDay="01-01"
   ,
   [ValidateScript({
      if (-Not ($_ | Test-Path) ){
          throw "File or folder does not exist" 
      }
      if (-Not ($_ | Test-Path -PathType Leaf) ){
          throw "The InputAPX argument must be a file. Folder paths are not allowed."
      }
      return $true
   })]
   [System.IO.FileInfo]$InputAPX=".\TEMPLATE.apx"
   ,
   [System.IO.FileInfo]$OutputAPX=".\RollingBacktests.apx"
   ,
   [ValidateNotNullOrEmpty()]
   [string]$OutputCSV=".\RollingBacktests - {0} - {1} to {2}.csv"
   ,
   [Switch]$NoCSV
   ,
   [Switch]$Quiet
)
#endregion

$DateFormatStr = "{0}-$MonthDay"  # YYYY-MM-DD

# instantiate Amibroker COM objects
$AB = New-Object -ComObject "Broker.Application"
$AD = $AB.AnalysisDocs()

# range of years to process
$years=$StartYear..$EndYear
foreach ($year in $years) {
   $FromDate = Get-Date ($DateFormatStr -f $year)
   $ToDate = ($FromDate).AddYears($StepYear)

   # break the loop if the FromDate is > EndYear
   if ($FromDate -gt (Get-Date ($DateFormatStr -f $EndYear))) {break}

   $FromDateStr = (Get-Date $FromDate -Format "yyyy-MM-dd")
   $ToDateStr = (Get-Date $ToDate -Format "yyyy-MM-dd")

   # read in the Input APX (template) file
   [xml]$template = Get-Content "$InputAPX"

   # Change the from and to dates
   $template.'Amibroker-Analysis'.General.FromDate="$FromDateStr"
   $template.'Amibroker-Analysis'.General.ToDate="$ToDateStr"

   # Get the formula basename
   $FormulaPath = $template.'AmiBroker-Analysis'.General.FormulaPath
   $FormulaName = (Get-Item "$FormulaPath").Basename

   # Save the Input APX (template) file to the Output APX (temporary) file
   $template.Save($OutputAPX)

   # Amibroker needs the full path to the OutputAPX
   $OutputAPX = Get-Item "$OutputAPX"

   # Run the script unless -whatif was specified (-verbose and -confirm are additional options)
   if ($PSCmdlet.ShouldProcess(
      "From-Date: $FromDateStr To-Date: $ToDateStr InputAPX $InputAPX","Rolling Backtest"
   ))
   {
      # Display this message regardless of Quiet setting
      "Rolling Backtest: From-Date: $FromDateStr To-Date: $ToDateStr InputAPX $InputAPX"

      # Set the CSV Output Filename
      $CSVOutputFile = $OutputCSV -f $FormulaName, (Get-Date $FromDate).Year, (Get-Date $ToDate).Year

      # Use the Amibroker COM object to run the backtest
      Try {
         $APX = $AD.Open($OutputAPX)
      }
      Catch {
         Throw "Unable to open $OutputAPX file."
      }

      if ($APX) {
         $rc=$APX.Run(2)
         if (-Not $quiet) {Write-Output "Run:$rc"}

         # Loop while the APX is running
         $ctr=0
         while ($APX.IsBusy()) {
            $ctr+=1
            if (-Not $quiet) {"{0,3}: APX is running" -f $ctr}
            Start-Sleep -seconds 1
         }

         # Export the trades list to CSV
         if (-Not $NoCSV) {$rc=$APX.Export($CSVOutputFile)}
         if (-Not $quiet) {Write-Output "Export CSV:$rc"}

         # Close the APX file
         $rc=$APX.Close()  # the Close method does not return a return code :(
         $rc=1
         if (-Not $quiet) {Write-Output "Close APX:$rc"}
         
         if (-Not $quiet) {Write-Output "Rolling Backtest Completed"}
      }
      else {
         Write-Error "Error occurred."  # should never happen
      }
   }
}

# Dispose of COM objects
if ($APX) {$rc=[System.Runtime.Interopservices.Marshal]::ReleaseComObject($APX)}
if ($AD)  {$rc=[System.Runtime.Interopservices.Marshal]::ReleaseComObject($AD)}
if ($AB)  {$rc=[System.Runtime.Interopservices.Marshal]::ReleaseComObject($AB)}

# Delete references to disposed COM objects
Remove-Variable APX,AD,AB -Force -ErrorAction SilentlyContinue -WhatIf:$false -Confirm:$false

# I've commented this out so I can debug the generated OutputAPX file but feel free to uncomment this if desired
#Remove-Item $OutputAPX

6 Likes

@LinusVanPelt, very well-written PowerShell script! :clap:

Thanks for sharing it with us.

1 Like

Big thanks for sharing this!