What am I doing wrong with Bar Replay backtest?

[mcescher]

Issue:

1. If I limit the number of allowable open positions, then Bar Replay portfolio backtest trades don't match the trade results of a full date-range portfolio backtest. If I don't constrain the number of open positions, then the Bar Replay backtest results match the results of a full date-range backtest.

2. When I limit the number of open positions (in my case to 12 in this example), if there is only one signal on a bar, then for that bar the result of the Bar Replay backtest will match the result of the full date-range backtest for that bar. In contrast, if multiple signals arise on a bar and PositionScore comes into play, that is when the result of the Bar Replay backtest may NOT match the result of the full date-range backtest for that bar.

Regarding #2 above, an excerpt of the Detail Log follows for the bar at 10:50:00. At this bar, the Bar Replay backtest goes short BA, whereas the full date-range backtest goes long MU. Prior to the 10:50:00 bar, the Bar Replay backtest trades matched the full date-range backtest trades.

Detail Log excerpt:
2021/04/13 10:50:00
Entry signals(score):MU=Buy(37.10079), BA=Short(-72.81805),
Exit signals:BA=Sell, MU=Cover,
Exit Long, BA, Price: 247.87, (Avg. exit pr. 247.87),
Shares: 10, Commission: 0.4, (Total comm.: 0.8), Profit: 9.3 (0.38 %), Entry rank:47.00171, Equity: 9999996, Fx rate: 1
Enter Long, MU, Price: 93.3181, Shares: 10, Commission: 0.4, Rank: 37.10079, Equity 9999996, Margin Loan: 466.7905, Fx rate: 1
12 Open Positions: , C (-10), , DD (-10), , TSLA (-10), , MRNA (+10), , IWM (+10), , IJR (+10), ,
NIO (+10), , PLUG (+10), , AMAT (-10), , XLI (+10), , XOP (-10), , MU (+10),
Market Value: 18113.89, Equity: 9999994.46, Cash: 9990989.79, Margin: -9109.22, Net Cash Balance: 9981880.57,

Backgrond:

• I'm using a 15 second database. 'Database time interval' is set to 15 seconds
• The Periodicity setting in the Analysis window is set to a 15 seconds
• The Bar Replay 'Step Interval' Is set to 15 seconds
• Watchlist has 23 symbols
• Running a simple moving avg crossover to generate many trades for testing
• Trades go long and short
• "SeparateLongShortRank" set to True

When I utilize Bar Replay...

• I'm using a repeating Exploration to execute a Batch, that in turn, runs a backtest. The repeating Exploration runs every 15 seconds. So essentially, I have a Bar Replay backtest that runs every 15 seconds.
• For Bar Replay I use static vars to keep track of the number of open positions and manage the limit of 12 positions.

Can anybody help me see what I'm doing wrong?

// #pragma maxthreads 1 ;

//Determine type of Analysis data during Scan, Exploration or Backtest
//==========================================================================================================
// 1 = Real time streaming
// 2 = Bar Replay
// 3 = Date-range backtest
//
if( ! IsNull( GetRTData( "Last" ) ) )
{
    // Repeating backtests with streaming data
    StaticVarSet( "_analysisType", 1 ); 
}
else
    if( GetPlaybackDateTime() > 0 )
    {
        // Bar Replay with repeating backtests
        StaticVarSet( "_analysisType", 2 ); 
    }
    else
    {
        // Full date-range backtest
        StaticVarSet( "_analysisType", 3 ); 
    }

_analysisType = StaticVarGet( "_analysisType" );


// Set configuration variables for run time
//==========================================================================================================

// Set which CBT backtest mode?  1 = backtestRegular  2 = backtestRegularRaw  3 = backtestRegularRawMulti
//--------------------------------------------------------------------------------------------------------
whichBacktestMode = 2 ;  // backtestRegular is the Amibroker default. If no setting is specified, backtestRegular applies.

StaticVarSet( "_backtestMode", whichBacktestMode ); // use this to know which backtest mode is in use.


// Trace vars
//-----------------------------------------------
tracePhase1Loop = 0;
tracePhase1SignalsBarReplay = 1 ;
tracePhase1SignalsLoop = 0 ;
traceNewBarTime = 0 ;


// Date, time & trading hours vars
//-----------------------------------------------
Buy = Short = Sell = Cover = 0 ;
BuyPrice = ShortPrice = SellPrice = CoverPrice = Close ;
dt =  DateTime();
dtStr = DateTimeToStr( LastValue( dt ), 3 );  // option 3 = YYYY-MM-DD HH:MM:SS format
tn = TimeNum();  // TimeNum examples... 093000 = 9:30 AM    110000 = 11:00 AM   133000 = 1:30 PM  154500 = 3:45 PM
bir = Status( "barinrange" );
myPeriodicity = Interval(); // bar duration in seconds
newBarTime = Ref( tn, -1 );
newBarTimeFlag = IIf( tn != Ref( tn, -1 ), 1, 0 );
myDayStr = Date();
myDayDT = StrToDateTime( myDayStr ) ;
myTime = StrFormat( "%02.0f:%02.0f:%02.0f", Hour(), Minute(), Second() );  // hh:mm:ss format

// Trading hours start time and end time
//---------------------------------------------------------------
startTimeTN = 093000; // start in TimeNum() HHMMSS format
endTimeTN = 155500;  // end in TimeNum() HHMMSS format

// Time OK trading hours date range
//---------------------------------------------------------------
timeOK = tn >= startTimeTN AND tn < endTimeTN;


// Set other vars section
//============================================================================================================

// Bar counter 
//--------------------------------------
rtBarCount = Nz( StaticVarGet( "_rtBarCount" ) );

// Other misc vars
//--------------------------------------
ticName = Name();
bi = BarIndex();
clearDBV = 0;


// Set Options
//=====================================================================================================

SetOption( "usecustombacktestproc", 0 );

SetOption( "AllowSameBarExit", 1 ); 

SetOption("HoldMinBars", 1 );

SetOption("SeparateLongShortRank", 1 ); 


// Configure ReverseSignalForcesExit
//-----------------------------------
oppositeSignalForcesExit = False;
SetOption( "ReverseSignalForcesExit", oppositeSignalForcesExit ); // True = reverse entry signal forces exit of existing trade


// Position Sizing & Initial Equity
//--------------------------------------
SetOption( "InitialEquity", 10000000 );
maxPositions = 12 ;
SetOption( "MaxOpenPositions", maxPositions );
SetOption( "MinShares", 10 );
SetOption( "MinPosValue", 0 );
SetPositionSize( 10, spsShares );
RoundLotSize = 10;
PositionScore = 100 - RSI();
// end position sizing


// Other Stuff
//--------------------------------------
SetTradeDelays( 0, 0, 0, 0 );
//SetOption( "MinShares", 50 );
SetOption( "PriceBoundChecking", False );
SetOption( "ActivateStopsImmediately", True );
SetOption( "AllowPositionShrinking", 1 );
SetOption( "AccountMargin", 50 );
SetOption( "CommissionMode", 3 );
SetOption( "CommissionAmount", 0.04 );


// This section controls which backtest mode will run based on the whichBacktestMode var set prior to this point
//============================================================================================================
if( whichBacktestMode == 1 )
{
    SetBacktestMode( backtestRegular );
    whichBacktestModeStr = "BacktestRegular" ;
}

if( whichBacktestMode == 2 )
{
    whichBacktestModeStr = "BacktestRegularRaw" ;
    SetBacktestMode( backtestRegularRaw );
}

if( whichBacktestMode == 3 )
{
    SetBacktestMode( backtestRegularRawMulti );
    whichBacktestModeStr = "BacktestRegularRawMulti" ;
}


// Initialization - Functions & Static vars for startup
//===========================================================================================================
//
// function createSymbolStaticVars( listNum, dt )
//-----------------------------------------------------------------------------------------------------------
// Function to create variables that store various parameters relatcreateSymbolStaticVars( listNum, dt)
// Create and initialize static vars for each symbol in the watchlist
// This code should only be be run once at the start of a realtime trading session
function createSymbolStaticVars( listNum )
{
    if( Status( "stocknum" ) == 0 )
    {
        _TRACE( "# Running createSymbolStaticVars... \n" );
        // Delete all static vars
        StaticVarRemove( "*" );

        // Get the first list in the Amibroker Watchlist window.  This will be List 0 by default, but you typically import
        // a watchlist into postion 0 so it will likely have a name other than List 0. The function is called with listnum = 0
        symlist = GetCategorySymbols( categoryWatchlist, listnum );
        symCount = StrCount( symlist , "," ) + 1;
        StaticVarSet( "_watchlistSymCount", symCount );

        for( i = 0; ( sym = StrExtract( symlist, i ) ) != ""; i++ )
        {
            //_TRACE( "# Symbol = " + sym );
            StaticVarSet( sym + "_prevPos", 0 );
            StaticVarSet( sym + "_prevPosDT", Null );  
        }
    }
    return;
}


function resetStaticVars()
{
    _TRACE( "# Executing resetStaticVars..." );
    //StaticVarSet( "InitializationDone", 0 );
    StaticVarSet( "startingDT", dt );
    StaticVarSet( "_rtBarCount", 0 );
    return;
}


// Run once via Nz( StaticVarGet("InitializationDone")
//----------------------------------------------------------------------------

if( Nz( StaticVarGet( "InitializationDone" ) ) == 0 )
{
    StaticVarSet( "InitializationDone", 1 );

    if( Status( "stocknum" ) == 0 )
    {
        //_TRACE( "# Running initialization code.  InitDone var = " + StaticVarGet( "InitializationDone" ) );

        // code for first execution goes below this line
        ////////////////////////////////////////////////

        // Create static vars for each symbol in the List 0 watchlist
        // List 0 is the first watchlist in the watchlist window
        createSymbolStaticVars( 0 );
        resetStaticVars() ;
        
        //Say( "hi, good morning", 0 );

        // The starting date/time of your APX
        StaticVarSet( "startingDT", dt );
      
    }
}  // end if( Nz( StaticVarGet( "InitializationDone" ) ) != 1 )


// Find out how many positions are open
//--------------------------------------
// During Bar Replay the position count will be used as a comparison to the maximum open positions value.
function zOpenPositionCount( listNum )
{
        // Get the first list in the Amibroker Watchlist window.  This will be List 0 by default, but you typically import
        // a watchlist into postion 0 so it will likely have a name other than List 0. The function is called with listnum = 0
        symlist = GetCategorySymbols( categoryWatchlist, listnum );
        zOpenPosCount = 0 ;

        for( i = 0; ( sym = StrExtract( symlist, i ) ) != ""; i++ )
        {
		  zOpenPosStatus = StaticVarGet(sym + "_prevPos" );

		  if ( zOpenPosStatus != 0  ) 
		     zOpenPosCount = zOpenPosCount + 1 ; 
		     //_TRACE("# "+ sym + " zOpenPosStatus = " + zOpenPosStatus + " 	zOpenPosCount = " + zOpenPosCount);
        }
return zOpenPosCount  ;      
}


// Moving Averages
//============================================================================================================
// Generate abunch of test signals
MA1 = 5;
MA2 = 100;  

ema1 = EMA( C, MA1 );
ema2 = EMA( C, MA2 );

maCrossBuy = Cross( ema1, ema2 );
maCrossSell = Cross( ema2, ema1 );
maCrossCover = maCrossBuy;
maCrossShort = maCrossSell;

refCloseOlder = -3 ;
refCloseNewer = -1 ;
maintainBuy = IIf( Ref( C, refCloseNewer ) >= Ref( C, refCloseOlder ), 1, 0 );
maintainSell = IIf( Ref( C, refCloseNewer ) <= Ref( C, refCloseOlder ), 1, 0 );

refMaCross = -6 ;

// Maintain the position state of each symbol...   open short = -1   no open position = 0   and open long = 1
//================================================================================================================
// previous position state...  open short = -1   no position = 0   open long = 1
_prevPos = StaticVarGet( ticName + "_prevPos" ); 
//
// the datetime that _prevPos was last changed
_prevPosDT = LastValue (StaticVarGet( ticName + "_prevPosDT" )); 
//
// the bar index at the time _prevPos was last changed
prevPosBar = Lookup(BarIndex(), _prevPosDT, 1);

_prevPos= IIf ( tn <= 093000, 0, _prevPos) ;
_prevPosDT= IIf ( tn <= 093000, Null, _prevPosDT) ;


// Buy & Sell Conditions
//=============================================================================
startRunDT = StaticVarGet( "startingDT" );
barsSinceStart = BarsSince( dt <= startRunDT );
dt2 = DateTimeToStr( LastValue(Ref(dt,0)) , 5);

// Get the openpositions count
zOpenPosCount = zOpenPositionCount( 0 ) ;

BC1 = Ref( maCrossBuy, refMaCross );
BC2 = maintainBuy;
BC3 = _prevPos != 1 ;
BC4 = IIf(zOpenPosCount < maxPositions, 1,
      IIf(zOpenPosCount == maxPositions AND _prevPos == -1, 1, 0));

SC1 = Ref( maCrossSell, refMaCross );
SC2 = maintainSell;
SC3 = _prevPos != -1 ;
SC4 = IIf(zOpenPosCount < maxPositions, 1, 		   
      IIf(zOpenPosCount == maxPositions AND _prevPos == 1, 1, 0));

Buy = IIf(bir AND timeOK AND BC1 AND BC2 AND BC3 AND BC4, 1, 0  ); 
Short = IIf( bir AND timeOK AND SC1 AND SC2 AND SC3 AND SC4, 1, 0 );

Buy = ExRem( Buy, Short );
Short = ExRem( Short, Buy );

Sell = Short;
Cover = Buy;

_prevPos = IIf(Buy, 1, IIf(Short, -1, LastValue(_prevPos) ));
_prevPosDT = IIf(Buy OR Short, dt, LastValue(_prevPosDT) );

StaticVarSet( ticName + "_prevPos", LastValue(_prevPos) );
StaticVarSet( ticName + "_prevPosDT", LastValue(_prevPosDT) );

InShort = Flip(Short, Cover);
InLong = Flip(Buy, Sell);
InTrade = InLong OR InShort;

// isEntryOrExit is useful if you export your results to a CSV file 
// use this var value to filter for all rows that have an entry or an exit signal 
isEntryOrExit = IIf(Buy OR Sell OR Short OR Cover , 1, 0);

// Update the open positions count
zOpenPosCount = zOpenPositionCount( 0 ) ;


if( tracePhase1SignalsBarReplay )
{
    if( LastValue( Buy ) OR LastValue( Sell ) )
    {
        _TRACE( "#, " + ticName + ", "  + DateTimeToStr( LastValue( dt ) ) + ", " + DateTimeToStr( LastValue( _prevPosDT ) ) + ", "
                + zOpenPosCount + ", " + Buy + ", " + Sell + ", " + Short + ", " + Cover + ", " + PositionScore + ", "
                + inLong + ", " + inShort + ", "
                + _prevPos + ", " + prevPosBar + ", " + bi + ", " + isEntryOrExit + ", " + Close );
    }
}

// Exploration stuff
//=============================================================================
if( Status( "action" ) == actionExplore )
{
    bir = Status( "barinrange" );

    //Filter = ( Buy OR Sell OR Short OR Cover);
    //Filter = ticName == "NIO" OR ticName == "AAPL" ;
    Filter = 1;

    //Filter = (Buy OR Sell OR Short OR Cover) AND tn >= startTimeTN AND tn < endTimeTN ;

    //Filter = eodExitLongFlag OR eodExitShortFlag ;
        
	AddColumn(_prevPosDT, "_prevPosDT", formatDateTime);
	AddColumn( Buy , "Buy" );
    AddColumn( Sell , "Sell" ); 
    AddColumn( Short , "Short" );
	AddColumn( Cover , "Cover" );	  
	AddColumn( inLong, "inLong" );
	AddColumn( inShort, "inShort" );
	AddColumn( inTrade, "inTrade" );
	AddColumn( positionScore, "positionScore" );
	AddColumn( highestPosScore, "highestPosScore" );
	AddColumn( _prevPos, "_prevPos" );
	AddColumn( _prevPosDT, "_prevPosDT" );
	AddColumn( prevPosBar, "prevPosBar" );
    AddColumn( bi, "barIndex" );
	AddColumn( isEntryOrExit, "isEntryOrExit" );
	AddColumn( Close, "Close" );
    
}


// Reset static vars section
//============================================================================================
trigger = ParamTrigger( "Reset Static Vars.......", "Click here to reset" );

trigger2 = ParamTrigger( "Create CSV header.......", "Click here to create CSV header" );

if( trigger )
{
    //bar = 0 ;
    _prevPos = 0;
    
    //Reset static vars
    createSymbolStaticVars( 0 );
    resetStaticVars();
    
    trigger = 0;
}


if( trigger2 )
{ 
    if( tracePhase1SignalsBarReplay )
    {
        _TRACE( "#, ticName, dt, _prevPosDT, zOpenPosCount, buy, sell, short, cover, posScore, inLong, inShort, prevPos, prevPosBar, bi, isEntryOrExit, Close " );
    }
    
    trigger2 = 0;
}


// Function to clear internal and external TRACE logs
//-----------------------------------------------------------------------------
function clearLogs()
{
    if( traceOther )
    {
        _TRACE( "# DBGVIEWCLEAR" ); // clears the external log file in the Microsoft Debugviewer app
        _TRACE( "!CLEAR!" ); // clears the internal log file in the Amibroker app
    }
}


// Run once code - various forms of 'Run once' code are in this section
//=============================================================================

// Run once via Status( "stocknum" ) == 0
//----------------------------------------------------------------------------
if( Status( "stocknum" ) == 0 )
{
    // Code in this section will run once at the start of each new bar
    // Code in this section is not applied to each symbol
    // run once code goes below this line
    //////////////////////////////////////////////////////////////////////////////////


    // play a sound at the beginning of each new bar
    // -----------------------------------------------
    // PlaySound( "c:\\windows\\media\\ding.wav" );


    // increment bar counter for real time processing
    //-------------------------------------------------
    rtBarCount = rtBarCount + 1;
    StaticVarSet( "_rtBarCount", rtBarCount );


    // send timestamp of each new bar to trace output
    //-------------------------------------------------- -
    if( traceNewBarTime )
    {
        //_TRACE( "# \n" );
        _TRACE( "# Running APX at timeNum = " + myTime + "  rtBarCount = " + rtBarCount + " \n" );
        //_TRACE( "# \n" );
    }


    // If the clearDBV is true, then call the clearLogs function
    // Is this useful?  Likely not necessary if trace statements are controlled by conditional flags?
    // -----------------------------------------------------------
    if( clearDBV )
    {
        clearLogs();
    }


}  // end run code once if( Status( "stocknum" ) == 0 )

//=============================================================================


if( Status( "actionEx" ) == actionExOptimizePortfolio )
{
    createSymbolStaticVars( 0 );
    resetStaticVars();
}

[Tomasz]

First thing to note is that Bar Replay is not really a backtest tool.

What Bar Replay does is to "CUT" the data at certain point (so data after "cutoff date" are not available). Then cutoff date is moved step by step effectively reproducing what you would see if data were coming on bar by bar basis.

That is all that Bar Replay does. It does not do any backtesting.

On the other hand when you are running PORTFOLIO backtest, AmiBroker simulates the entire ACCOUNT, with its limitations and constraints. Your account during portfolio backtesting has limited buying power and symbols are selected based on position score, which means that some of trades are NOT taken (if you run out of funds), i.e. some signals are skipped. See detailed information in the manual:
http://www.amibroker.com/guide/h_portfolio.html

When you are doing Bar Replay and add your own code that does some kind of "backtesting by hand", there are NO account restrictions, no constraints, so you never run out of funds and you open ALL positions taking ALL signals. That is NOT what happens in real life and NOT what happens in portfolio backtesting.

See also:

Scan vs Exploration vs Backtest - summary of differences

INTRO The post summarizes in tabular form the differences between different Analysis modes namely: Scan, Exploration and Backtest. First and foremost thing to understand is that these three modes serve different and distinct purposes and they by their nature are meant to display different things. SCAN The Scan is meant as a quick tool to show raw, unprocessed, non-delayed signals as they are present in Buy/Sell/Short/Cover arrays. Scan does not do ANY processing. Output of scan is not custom…

[mcescher]

Thanks for the feedback Tomasz. After further testing I realized a few things:

1. When my open position counter reached an amount equal to one less than the max open positions limit, my phase 1 code would evaluate signals for each symbol in my watchlist and would treat the first viable signal as an entry, even if other symbols had better position scores, then the position counter would be incremented by 1. At that point, the counter was equal to the max open position amount and my Buy/Short conditions would skip any other symbols in the watchlist even if they had a better position score.

2. Removing my Buy/Short conditions that included the open position counter, and moving/modifying that logic to phase 2 in a CBT version helped me make significant progress.

Thanks again for the comments and references. Much appreciated. Even if Bar Replay was not intended for backtesting, I learned more things about Amibroker while trying it in a backtesting scenario.