Data holes: Long only strategy is taking short trades

Hello All,
I'm quite new to Amibroker and still learning. This forum and the Amibroker tutorials has been excellent in getting me started.

I am however now stuck with an Issue where I need some guidance on how to manage data.

Background: i'm coding a Long only strategy based on relative strength of a stock against a benchmark index. I want to buy the top 10 ranked stocks, and hold them until they have dropped out of the Top 20 ranks. The ranks should be evaluated every week on daily timeframe data.

Problem: As the title states, while in most rotations the correct stocks are being rotated in/out based on the ranks, during some rotations the strategy exits longs and shorts some stocks.

I tested this in a narrow date range to better show the problem.
The stocks that the code is shorting for e.g. on 8-Nov-2010 are from the previous trading day i.e. 05-Nov-2010 where my database doesn't have the data for these stocks for that day.

Exploration that explains this

Please let me know how I can work around these data holes. BTW, I do have Pad & align turned on and the the bar exists in the reference symbol.

My code below for reference in case there's any workaround that I can implement in the code itself.

// set all the backtest settings in AFL code so that its standardized across backtests.
SetFormulaName("Rotational Weekly");
SetBacktestMode(backtestRotational);


// 1 ###### BACKTESTER SETTINGS - 1. GENERAL TAB 
PosQty = Optimize("PosQty", 10, 5, 20, 1);
WorstRank = Optimize("WorstRank", 20, 10, 40, 5);

SetOption("InitialEquity", 1000000); // 
SetOption("MinShares", 1); // atleast buy 1 share, no fractional shares
SetOption("MinPosValue", 0);
SetOption("FuturesMode", False);
SetOption("AllowPositionShrinking", True);
SetOption("ActivateStopsImmediately", True);
SetOption("ReverseSignalForcesExit", False);
SetOption("AllowSameBarExit", True);
RoundLotSize = 1;
SetOption("CommissionMode", 1);  //1= % of trade; 2 - $ per trade; 3 - $ per share/contract
SetOption("CommissionAmount", 0.07); // STT = 0.0005 for each buy & sell; Transaction charges NSE: 0.0000325; Fixed DP Charge on Sell: Rs. 15.93 - just setting to 0.07 on higher side
SetOption("AccountMargin", 100); // AccountMargin - account margin requirement 100 = no margin 
SetOption("UsePrevBarEquityForPosSizing", True); 

// 2 ###### BACKTESTER SETTINGS - 2. TRADES TAB  - Trade timing
BuyPrice = SellPrice = Open;
SetTradeDelays(1,1,1,1);	// Trades are entered 1 day after signal is generated i.e. next week open as coded

// 5 ###### BACKTESTER SETTINGS - 5. PORTFOLIO TAB - position sizing etc
SetOption("MaxOpenPositions", PosQty);

// Limit trade size as % - use 10 for live trading
// check the box to "Disable trade size limit..."
SetOption("UsePrevBarEquityForPosSizing", True);
SetOption("UseCustomBacktestProc",  False);
SetOption("WorstRankHeld", WorstRank);
SetOption( "MaxOpenLong", PosQty );
SetPositionSize(100/PosQty,spsPercentOfEquity); // calculate position size as % of equity automatically based on max # of stocks held
//SetOption("SeparateLongShortRank", True ); // not needed?

//######################## Strategy Code ########################
// watchlist should contain all symbols included in the test
wlnum = GetOption( "FilterIncludeWatchlist" );
List = CategoryGetSymbols( categoryWatchlist, wlnum ) ;
ListQty = StrCount(List, ",") + 1; 

// Index optimizer 
RSIndex = "^CRSLDX";
RSIndexMA = 34;
IndexClose = 0;
IndexMA = 0;

// Iterate through all stocks in watchlist to calculate RS at each periodicity
// Relative Strength calculation of stocks
// This code must be run in Analysis Engine with Daily periodicity.
if( Status( "stocknum" ) == 0 )
{
	// Clear the static vars from last run, not required, but safer
	StaticVarRemove("RS*");
	StaticVarRemove("RankRS*");
		
	IndexRSRatio55 = 0;

	//######################## Get Index values to calculate relative strength ########################
	SetForeign(RSIndex); //switch away to Index for calculating Index ratio. Done before the loop so its only called once
		IndexRSRatio55 = C/Ref(C, -55);
	RestorePriceArrays(); //switch back to current symbols
		
	// Generate the raw RS score for every stock and store in a static var
	for (n = 0; (Symbol = StrExtract(List, n)) != "";  n++)
	{ 
		SetForeign (Symbol,0); 
			
			StockRS55 = 0;
			
			StockRS55 = ((C/Ref(C,-55)) / IndexRSRatio55 ) -1; // Last 3 months RS
	
			// Set static var values referenced to the current symbol in the loop
			StaticVarSet("RS_"  +  Symbol, 1000+StockRS55);
			StaticVarSet("RSValue"  +  Symbol, StockRS55);
			_TRACE(Date() +" Ticker="+Symbol +" RSValue="+ NumToStr(StockRS55) );
	
		RestorePriceArrays();
	}

	// rank the stocks
	StaticVarGenerateRanks("Rank", "RS_", 0, 1234);
}

Rank  = StaticVarGet ("RS_" +  Name());

EndOfWeek = DayOfWeek() > Ref( DayOfWeek(), 1 ); //looks ahead so can't use in realtime

PositionScore =  IIf(EndOfWeek, Rank, scoreNoRotate);
//PositionScore = Rank;

// read ranking for debugging
/*
for (n = 0; (Symbol = StrExtract(List, n)) != "";  n++) 
 { 
    //_TRACE("Rank="+ NumToStr(n) +"-"+ Symbol);
    _TRACE(Date() +" Ticker="+Symbol +" RSValue="+ NumToStr(StaticVarGet("RSValue"+Symbol)) +" Rank="+ NumToStr(StaticVarGet("RankRS_"+Symbol)) );
 } 
*/

SetSortColumns(3);

//######################## Exploration Section ########################
if( Status( "Action" ) == actionExplore )
{
Filter = (C > 1.0);

AddColumn(Close, "Price", 1.2);
AddColumn(Rank, "Rank", 1.3);
AddColumn(MA(Close, 20), "20 SMA", 1.3);

SetSortColumns(2, -4);
}

Please read the documentation
https://www.amibroker.com/guide/h_portfolio.html

It says

AmiBroker will use the absolute value of PositionScore variable to decide which trades are preferred

The symbols marked with RED have larger absolute value of score, than symbols you have marked with BLUE.

The absolute value is the value WITHOUT sign.

Also, in Rotational Mode, the SIGN of PositionScore decides which side is taken. NEGATIVE value of PositionScore means SHORT trade.

Again, please read the documentation
https://www.amibroker.com/f?enablerotationaltrading

Quoting the documentation:

The score (PositionScore) for all securities is calculated first. Then all scores are sorted according to absolute value of PositionScore. Then top N are choosen to be traded. N depends on available funds and "max. open positions" setting. Backtester successively enters the trades starting from highest ranked security until the number of positions open reaches "max. open positions" or there are no more funds available. The score has the following meaning:

  • higher positive score means better candidate for entering long trade
  • lower negative score means better candidate for entering short trade
  • the score of zero means no trade (exit the trade if there is already open position on given symbol)
  • the score equal to scoreNoRotate constant means that already open trades should be kept and no new trades entered
  • the score equal to scoreExitAll constant causes rotational mode backtester to exit all positions regardless of HoldMinBars. Note that this is global flag and it is enough to set it for just any single symbol to exit all currently open positions, no matter on which symbol you use scoreExitAll (it may be even on symbol that is not currently held). By setting PositionScore to scoreExitAll you exit all positions immediatelly regardless of HoldMinBars setting
1 Like

Hi Tomasz,

Thank you for the response. I understand how PositionScore works, my question is how can I workaround these types of data holes, considering that I already have Pad & Align enabled and the reference symbol has data for this day i.e. 5-Nov-2010.

Is there any way in the rotational code where I could or ignore or suppress negative numbers from being considered in the ranking for e.g. wrapoing the StaticVarSet() calls within an If condition that allows only positive values of StockRS55.

Could that be possible or recommended?

// Iterate through all stocks in watchlist to calculate RS at each periodicity
if( Status( "stocknum" ) == 0 )
{
	// Clear the static vars from last run, not required, but safer
	StaticVarRemove("RS*");
	StaticVarRemove("RankRS*");
		
	IndexRSRatio55 = 0;

	//######################## Get Index values to calculate relative strength ########################
	SetForeign(RSIndex); //switch away to Index for calculating Index ratio. Done before the loop so its only called once
		IndexRSRatio55 = C/Ref(C, -55);
	RestorePriceArrays(); //switch back to current symbols
		
	// Generate the raw RS score for every stock and store in a static var
	for (n = 0; (Symbol = StrExtract(List, n)) != "";  n++)
	{ 
		SetForeign (Symbol,0); 
			
			StockRS55 = 0;
			
			StockRS55 = ((C/Ref(C,-55)) / IndexRSRatio55 ) -1; // Last 3 months RS

//##!! wrap these below lines of code inside an If condition which only adds positive values to the StaticVars collection
			// Set static var values referenced to the current symbol in the loop
			StaticVarSet("RS_"  +  Symbol, 1000+StockRS55);
			StaticVarSet("RSValue"  +  Symbol, StockRS55);
			_TRACE(Date() +" Ticker="+Symbol +" RSValue="+ NumToStr(StockRS55) );
//##!! 	
		RestorePriceArrays();
	}

	// rank the stocks
	StaticVarGenerateRanks("Rank", "RS_", 0, 1234);
}

To try and workaround the issue of data holes, I tried to assign a positive value to the variable used for PositionScore.

Basically now the code loop snippet looks like this

// Generate the raw RS score for every stock and store in a static var
	for (n = 0; (Symbol = StrExtract(List, n)) != "";  n++)
	{ 
		SetForeign (Symbol,0); 
			
			StockRS55 = 0;
			
			StockRS55 = ((C/Ref(C,-55)) / IndexRSRatio55 ) -1; // Last 3 months RS
	
			if(SelectedValue(StockRS55) > 0.0)
			{
				staticValue=1000+StockRS55;
				_TRACE( NumToStr(n)+": "+ Date() +" In Loop: Ticker="+ Symbol +" StockRS55="+ NumToStr(SelectedValue(StockRS55)) +" staticValue="+ NumToStr(staticValue) );
			}
			else
				staticValue=1000.001;
			
			
			// Set static var values referenced to the current symbol in the loop
			StaticVarSet("RS_"  +  Symbol, staticValue);
			StaticVarSet("RSValue"  +  Symbol, StockRS55);
			
	
		RestorePriceArrays();
	}


	// rank the stocks
	StaticVarGenerateRanks("Rank", "RS_", 0, 1234);
}

Rank  = StaticVarGet ("RS_" +  Name());
//_TRACE(Date() +" Out: Ticker="+ Name() +" Rank="+ StaticVarGet ("RS_" +  Name()));

EndOfWeek = DayOfWeek() > Ref( DayOfWeek(), 1 ); //looks ahead so can't use in realtime

PositionScore =  IIf(EndOfWeek, Rank, scoreNoRotate);
_TRACE(Date() +" Ticker="+ Name() +" RS_="+ NumToStr(StaticVarGet("RS_"+Name())) +" PositionScore="+PositionScore +" Rank="+ NumToStr(StaticVarGet("RankRS_"+Name())));

However even now while the PositionScore should have only Positive values in the array assigned to it, it still goes short on the same trades as earlier.
So forcing a positive number has had no effect on the strategy going long only. Is the PositionScore looking at some number other than the array assigned to it?

Below is the log extract from _Trace() after PositionScore is assigned. Note that all PositionScore values are positive - only top 20 ranked are shown in screenshot.

But the strategy still goes short on 7 stocks which I cannot understand why.

I am probably missing something but I have coded this based on my understanding from the documentation of PositionScore (absolute values) and how long/short positions are taken (based on the socre being positive or negative). So shouldn't the PositionScore having only positive values result in long only positions?

Any help or guidance in helping me understand this better is much appreciated! @Tomasz @fxshrat

Thanks in advance!

latest version full code is below for reference


// set all the backtest settings in AFL code so that its standardized across backtests.
SetFormulaName("Rotational Weekly");
SetBacktestMode(backtestRotational);


// 1 ###### BACKTESTER SETTINGS - 1. GENERAL TAB 
PosQty = Optimize("PosQty", 10, 5, 20, 1);
WorstRank = Optimize("WorstRank", 20, 10, 40, 5);

SetOption("InitialEquity", 1000000); // 
SetOption("MinShares", 1); // atleast buy 1 share, no fractional shares
SetOption("MinPosValue", 0);
SetOption("FuturesMode", False);
SetOption("AllowPositionShrinking", True);
SetOption("ActivateStopsImmediately", True);
SetOption("ReverseSignalForcesExit", False);
SetOption("AllowSameBarExit", True);
RoundLotSize = 1;
SetOption("CommissionMode", 1);  //1= % of trade; 2 - $ per trade; 3 - $ per share/contract
SetOption("CommissionAmount", 0.07); // STT = 0.0005 for each buy & sell; Transaction charges NSE: 0.0000325; Fixed DP Charge on Sell: Rs. 15.93
SetOption("AccountMargin", 100); // AccountMargin (in old versions it was 'MarginRequirement') - account margin requirement 100 = no margin 
SetOption("UsePrevBarEquityForPosSizing", True); // UsePrevBarEquityForPosSizing - Affects how percent of current equity position sizing is performed.
												 // False (default value) means: use current (intraday) equity to perform position sizing,
												 // True means: use previous bar closing equity to perform position sizing 


// 2 ###### BACKTESTER SETTINGS - 2. TRADES TAB  - Trade timing
BuyPrice = SellPrice = Open;
//SetTradeDelays(1,1,1,1);	// Trades are entered 1 day after signal is generated i.e. next day open
SetTradeDelays(0,0,0,0);

// 5 ###### BACKTESTER SETTINGS - 5. PORTFOLIO TAB - position sizing etc
SetOption("MaxOpenPositions", PosQty);
SetOption("WorstRankHeld", WorstRank);
SetOption( "MaxOpenLong", PosQty );
SetOption( "MaxOpenShort", PosQty );
SetPositionSize(100/PosQty,spsPercentOfEquity); // calculate position size as % of equity automatically based on max # of stocks held
//SetOption("SeparateLongShortRank", True ); // not needed?

// Limit trade size as % - use 10 for live trading
// check the box to "Disable trade size limit..."
SetOption("UsePrevBarEquityForPosSizing", True);
SetOption("UseCustomBacktestProc",  False);


// 6 ###### BACKTESTER SETTINGS - 6. WALK FORWARD TAB

// Code
// Relative Strength ranking of stocks
// This code must be run in Analysis Engine with Daily periodicity.

// watchlist should contain all symbols included in the test
wlnum = GetOption( "FilterIncludeWatchlist" );
List = CategoryGetSymbols( categoryWatchlist, wlnum ) ;
ListQty = StrCount(List, ",") + 1; 

// Index optimizer 
RSIndex = "^CRSLDX";
RSIndexMA = 34;
IndexClose = 0;
IndexMA = 0;

// Iterate through all stocks in watchlist to calculate RS at each periodicity
if( Status( "stocknum" ) == 0 )
{
	// Clear the static vars from last run, not required, but safer
	StaticVarRemove("RS*");
	StaticVarRemove("RankRS*");
		
	IndexRSRatio55 = 0;

	//######################## Get Index values to calculate relative strength ########################
	SetForeign(RSIndex); //switch away to Index for calculating Index ratio. Done before the loop so its only called once
		IndexRSRatio55 = C/Ref(C, -55);
	RestorePriceArrays(); //switch back to current symbols
		
	// Generate the raw RS score for every stock and store in a static var
	for (n = 0; (Symbol = StrExtract(List, n)) != "";  n++)
	{ 
		SetForeign (Symbol,0); 
			
			StockRS55 = 0;
			
			StockRS55 = ((C/Ref(C,-55)) / IndexRSRatio55 ) -1; // Last 3 months RS
	
			if(SelectedValue(StockRS55) > 0.0)
			{
				staticValue=1000+StockRS55;
				_TRACE( NumToStr(n)+": "+ Date() +" In Loop: Ticker="+ Symbol +" StockRS55="+ NumToStr(SelectedValue(StockRS55)) +" staticValue="+ NumToStr(staticValue) );
			}
			else
				staticValue=1000.001;
			
			
			// Set static var values referenced to the current symbol in the loop
			StaticVarSet("RS_"  +  Symbol, staticValue);
			StaticVarSet("RSValue"  +  Symbol, StockRS55);
			
	
		RestorePriceArrays();
	}

	// rank the stocks
	StaticVarGenerateRanks("Rank", "RS_", 0, 1234);
}

Rank  = StaticVarGet ("RS_" +  Name());
//_TRACE(Date() +" Out: Ticker="+ Name() +" Rank="+ StaticVarGet ("RS_" +  Name()));

EndOfWeek = DayOfWeek() > Ref( DayOfWeek(), 1 ); //looks ahead so can't use in realtime

PositionScore =  IIf(EndOfWeek, Rank, scoreNoRotate);
//PositionScore = Rank;

_TRACE(Date() +" Ticker="+ Name() +" RS_="+ NumToStr(StaticVarGet("RS_"+Name())) +" PositionScore="+PositionScore +" Rank="+ NumToStr(StaticVarGet("RankRS_"+Name())));

// read ranking for debugging
/*
for (n = 0; (Symbol = StrExtract(List, n)) != "";  n++) 
 { 
    //_TRACE("Rank="+ NumToStr(n) +"-"+ Symbol);
    _TRACE(Date() +" Ticker="+Symbol +" RSValue="+ NumToStr(StaticVarGet("RSValue"+Symbol)) +" Rank="+ NumToStr(StaticVarGet("RankRS_"+Symbol)) );
 } 
*/

SetSortColumns(3);

//######################## Exploration Section ########################
if( Status( "Action" ) == actionExplore )
{
Filter = Rank > 0;

AddColumn(Close, "Price", 1.2);
AddColumn(Rank, "Rank", 1.3);
AddColumn(MA(Close, 20), "20 SMA", 1.3);

SetSortColumns(2, -4);
}

First you would need to simplify your code. It is way too long for what it is supposed to to.

To implement ranking, you don't need static variables at all.

Ranking is built-in feature and is done by backtester all out of the box.

Instead of using loops and calling StaticVarGenerateRanks
why don't you just assign the score directly to PositionScore. Remove 99% of the code that you have now and just do all ranking in SINGLE LINE OF CODE

PositionScore =  ROC( C, 55 ) / ROC( Foreign( RSIndex, "C" ), 55 );

All without this if( Status("stocknum") == 0 { ... } part

And just use "Detailed report" mode and you will see all ranking done automatically without ANY extra code.

And please do read the manual:
https://www.amibroker.com/guide/h_ranking.html

All you need is "FIRST KIND OF RANKING" as documented

The first kind of ranking is performed automatically if your trading system formula defines PositionScore variable. You can use PositionScore variable to decide which trades should be entered if there are more entry signals on different securities than maximum allowable number of open positions or available funds. In such case AmiBroker will use the absolute value of PositionScore variable to decide which trades are preferred. For the details about ranking functionality during backtesting see Portfolio Backtester tutorial.

1 Like

I will definitely simplify the code and do as you said.

After going through documentation and forum answers my understanding of the relative strength calculation in strategy became that the third kind of ranking is what is required in this situation.

Thank you for pointing me in the right direction on how to code it, you are right, only the First kind of ranking is required for the simple strategy that I am attempting.

@Tomasz, I did as you guided, and the simple way to do the PositionScore with the First kind of ranking, resolved the problems.

Now I'm substituting any data holes for the relative strength calculation with a small enough positive number, so that those symbols don't get traded, but also so that no short positions are opened like earlier and it's working as intended from what I can see, will test further.

Thank you very much for your time to point me in the right direction! This has really helped.