Ranking sometimes not working (when you have holes in data)

Hi all,

I have some ranking code that I intend to use to downsize the stock selection universe. In general it works, but sometimes it's not producing any trade. For example, this code works as expected:

if (Status( "stocknum" ) == 0) {
	for (i = 0; ( symbol = StrExtract( watchlist, i ) )  != "";  i++) {
		SetForeign(symbol);

		rank = ROC(C, 20);

		RestorePriceArrays();
		StaticVarSet("values" + symbol, rank);
	}
			
	StaticVarGenerateRanks("rank", "values", 0, 1224 ); 
}

// ... and later, only include symbols from top 200 rank
rank = StaticVarGet("rankvalues" + Name());
Buy = Buy AND rank <= 200;

However, if I only replace the ranking formula by something like this (rest being the same)

rank = ATR(20) / C;

I get no trades. Anyone has any idea why this happens? It's strange, because the same ranking formulas work as expected when using them in PositionScore, but when doing custom ranking some of them don't work as expected.

Thank you!

I don’t immediately see any errors in what you’ve shown here. Have you tried using an Exploration to display the ranking values being retrieved by StaticVarGet()?

Do you reset your static variables before the above code?

See an example in:
https://www.amibroker.com/guide/h_ranking.html

StaticVarRemove( "values*" );
// your code ....

Hi again,

I izolated a piece of code, now another problem appeared. Seems like sometimes the ranking condition is now respected. I'm puzzled now, I'm running the backtest over weekly timeframe and trying to allow only top 50 ranked symbols. I'm using the Norgate Alpha database. Here's the code:

// === Universe, equity, 
// position sizing, stop loss ====
universe = "$RUA";
initial_equity = 50000;
positions = 10;
sl_initial_trail = 0.6;

top_ranking = 50; // if > 0, then consider only top X ranked symbols

// === BO channel ===
channel_size = 20;

// === Delays ===
SetTradeDelays(1,1,1,1); 

// === Position sizing and commission ===
PosSize = (100/positions);
SetOption("Initialequity", initial_equity);
SetOption("MaxOpenPositions", positions);
SetOption("MaxOpenLong", positions);
SetPositionSize(PosSize, spsPercentOfEquity);
SetOption("MinShares", 1);

if (top_ranking > 0) {
	// watchlist should contain all symbols included in the test
	watchlist = CategoryGetSymbols(categoryWatchlist, GetOption( "FilterIncludeWatchlist" )) ;

	if (Status( "stocknum" ) == 0) {
		// cleanup variables created in previous runs (if any)
		StaticVarRemove("rank*");
		StaticVarRemove("values*");
		
		for (i = 0; ( symbol = StrExtract( watchlist, i ) )  != "";  i++) {
			SetForeign(symbol);
			
			// rank = ROC(C, 52);
			rank =  ATR(20) / C;
			RestorePriceArrays();
			
			//_TRACE(NumToStr(rank));
			
			StaticVarSet("values" + symbol, rank);
		}
		
		StaticVarGenerateRanks("rank", "values", 0, 1224 ); 
	}
}

channel = Ref(HHV(C, channel_size), -1); 
Buy = C >= channel;

if (top_ranking > 0) {
	rank = StaticVarGet("rankvalues" + Name());
	_TRACE(NumToStr(rank));
	Buy = Buy AND rank <= top_ranking;
}

PositionScore = ROC(20);

BuyPrice = Open; // backtesting
SellPrice = Open; // backtesting
Sell = 0;

DefaultStopLevel = sl_initial_trail;
trailARRAY = Null;
trailstop = 0;

for(i = 1; i < BarCount; i++){
	trail_point = H[i];
	if(trailstop == 0 AND Buy[i]){
		trailstop = trail_point * DefaultStopLevel;
	}else{
		Buy[i] = 0; // remove excess buy signals
	}

	if(trailstop > 0){
		trailARRAY[i] = trailstop;
    }	
    
	if(trailstop > 0 AND Close[i] < trailstop){
		Sell[i] = 1;
		trailstop = 0;
	}
}

Buy = ExRem (Buy, Sell);
Sell = ExRem (Sell, Buy);

// custom backtester, only to display rank
SetCustomBacktestProc(""); 
if( Status("action") == actionPortfolio ) 
{ 
    bo = GetBacktesterObject(); 

    bo.Backtest(true); // run default backtest procedure 

     // iterate through closed trades
    for (trade = bo.GetFirstTrade( ); trade; trade = bo.GetNextTrade())
    {
		// because of trade delays
		entryBar = Lookup(BarIndex(), trade.entrydatetime, 0) - 1;
        
        val = StaticVarGet("rankvalues" + trade.Symbol);
        trade.AddCustomMetric( "Rank", val[entryBar]);
        val = StaticVarGet("values" + trade.Symbol);
        trade.AddCustomMetric( "Rank Value", val[entryBar]);
    }

    // iterate through open positions
    for ( trade = bo.GetFirstOpenPos( ); trade; trade = bo.GetNextOpenPos( ))
    {
		// because of trade delays
		entryBar = Lookup(BarIndex(), trade.entrydatetime, 0) - 1;
        
        val = StaticVarGet("rankvalues" + trade.Symbol);
        trade.AddCustomMetric( "Rank", val[entryBar]);
        val = StaticVarGet("values" + trade.Symbol);
        trade.AddCustomMetric( "Rank Value", val[entryBar]);
    }

    // generate trade list
    bo.ListTrades( );
} 

I attached an image with the backtesting ranking, showing that symbols with ranking > 50 were accepted.

image

1 Like

Ok I found the cause, it’s because some symbols were delisted/listed at different times and data was missing. At least that’s what I found around while searching for the problem.

So I checked “Pad and align all data to reference symbol” and rankings seem used properly now. But I wonder if this is a realistic solution? What does checkbox? Isn’t including in my backtest some symbols that woudn’t be included otherwise? Is there a solution so I exclude all symbols that have no data/wrong data when using ranks?

Thank you.

1 Like

And last try, now seems it’s working ok. The main problem was the exclusion of delisted symbols. “Pad and align…” checkbox doesn’t have much importance as far as I can see.

For Norgate Alpha users, this is the code that I use:

SetForeign(symbol);

rank = ROC(C, 20); // your ranking routine
OnLastTwoBarsOfDelistedSecurity = BarIndex() >= (LastValue(BarIndex()) -1) AND !IsNull(GetFnData("DelistingDate"));
rank = IIf(C > 0 AND V > 0 AND IsIndexConstituent(universe) AND NOT OnLastTwoBarsOfDelistedSecurity, rank, -10000);

RestorePriceArrays();
StaticVarSet("values" + symbol, rank);

From http://www.amibroker.com/guide/w_settings.html

Pad and align to reference symbol
When this is turned on, all symbols’ quotes are padded and aligned to reference symbol. Note: by default, this setting is OFF. Use responsibly. It may slow down backtest/exploration/scan and introduce some slight changes to indicator values when your data has holes and holes are filled with previous bar data. The feature is intended to be used when your system uses general market timing (generates global signals based on data and/or indicators calculated using Foreign from ‘reference’ symbol) or when you are creating composites out of unaligned data. Note: if reference symbol does not exist, data won’t be padded.

Maybe you should clean up your database (see the @Tomasz answer)

or if you want to be more selective do a procedure like the one described here:

This could also be used to create a watchlist for symbols to EXCLUDE:

Search here in the forum works pretty well!

1 Like

Thanks beppe, the actual solution was along the same lines. I posted the solution to my data just before you replied.

One has to realize obvious fact that ANY kind of ranking REQUIRES data to be aligned (i.e. without holes).

2 Likes