Confirmation I am using Norgate historical constituents correctly

I looked on Norgate's website to learn what to put into my AFL to ensure I am getting an accurate view of historical constituents, however, I'm getting wildly different results in a strategy I'm working on when comparing SPX Current and Past (and using NorgateIndexConstituentTimeSeries("$SPX") vs. SPX (without the Norgate call) over the last 2 years (where most tickers haven't changed).

The huge difference is making me question if I have done everything correctly with respect to looking at Norgate data. I've recreated a simpler version of the strategy (which doesn't show as big of a difference) to show mostly just the Norgate portion to help illustrate this. I'm hoping that someone can confirm these are the right things to put in my code for this use case.

  1. At the top of my AFL, I should put the following to enable the calls to the Norgate functions:
#include_once "Formulas\Norgate Data\Norgate Data Functions.afl"
  1. I should define the following variables to avoid buying delisted stocks:
OnSecondLastBarOfDelistedSecurity = !IsNull(GetFnData("DelistingDate")) AND (BarIndex() == (LastValue(BarIndex()) -1) OR DateTime() >= GetFnData("DelistingDate") ) ;
OnLastTwoBarsOfDelistedSecurity = !IsNull(GetFnData("DelistingDate")) AND (BarIndex() >= (LastValue(BarIndex()) -1) OR DateTime() >= GetFnData("DelistingDate") );
  1. As part of my Buy criteria, I should add the following to ensure I'm not buying a stock that is about to be deslisted:
AND NOT OnLastTwoBarsOfDelistedSecurity
  1. To ensure I am looking into the historical constituents of the indices I also add this to the Buy criteria and to use "SPX Past and Current" and "NDX Past and Current" watchlists in my Filter.
AND (NorgateIndexConstituentTimeSeries("$SPX") OR NorgateIndexConstituentTimeSeries("$NDX"));
  1. And finally, I am adding this to my Sell logic to ensure I am getting out of stocks before they are delisted:
Sell = Sell OR OnSecondLastBarOfDelistedSecurity;

Do I have all of this right or am I missing anything to accurately pull historical index constituents and avoiding delisted stocks when running strategies?

Here's all of this in the context of a simplified strategy:

/*
NORGATE TEST
This strategy is just a simple strategy to ensure that the proper hooks to Norgate historical data are correct.
*/

// This should be included at the top of the AFL
#include_once "Formulas\Norgate Data\Norgate Data Functions.afl"

// ==========================================================================================
// === TRADING SYSTEM PARAMETERS ===
// ==========================================================================================
SetOption("InitialEquity", 100000);
SetTradeDelays( 0, 0, 0, 0 ); // 1 - Delays the trades to the next day, 0 - Trades same day via stops
SetOption("AllowPositionShrinking", True);
SetOption("AccountMargin",100); // 100 = Cash account, 75 = Using half available margin
SetOption("MaxOpenPositions", 20);
RoundLotSize = 1;  // Disallow fractional share purchasing


// ==========================================================================================
// === RISK MANAGEMENT & POSITION SIZING ===
// ==========================================================================================
// Initial (MAX) stop loss
SetOption("ActivateStopsImmediately", False);
stopLoss = 7; // Change this to a small number to experience the issue with trade entries being ignored
ApplyStop(stopTypeLoss, stopModePercent, stopLoss, ExitAtStop = 1);

// Position size based on risk and max stop loss
PositionRisk = 1;  
PosSize = 100 * PositionRisk / stopLoss;
SetPositionSize(PosSize, spsPercentOfEquity);
         

// ==========================================================================================
// === PLOT CHART & INDICATORS ===
// ==========================================================================================
// Chart
SetChartOptions(0,chartShowArrows|chartShowDates);
_N(Title = StrFormat("{{NAME}} - " + FullName() + " | {{INTERVAL}} {{DATE}}: Open %g, Hi %g, Lo %g, Close %g (%.1f%%) {{VALUES}}", O, H, L, C, SelectedValue( ROC( C, 1 ) ) ));
Plot( C, "Close", ParamColor("Color", colorDefault ), styleNoTitle | ParamStyle("Style") | GetPriceStyle() ); 

// SMAs
sma20 = ma(Close, 20);
sma200 = ma(Close, 200);
Plot(sma20,"20MA", color = colorGrey50, styleNoTitle);
Plot(sma200,"150MA", color = colorWhite, styleNoTitle);

// Donchian Channel
pdsU=4;  // High channel lookback
pdsL=5;  // Low channel lookback

DonchianUpper =HHV(H,pdsU);
DonchianLower = LLV(L,pdsL);

Plot(DonchianUpper,"DU",color = colorBlueGrey,styleDashed);
Plot(DonchianLower,"DL",color = colorBlueGrey, styleDashed);

// ==========================================================================================
// === FILTERING AND SCREENING CRITERIA ===
// ==========================================================================================
// Set other non-chart visible variables
vol = MA(Volume, 20); 

// Determine filtering criteria
Filter = 
  Close > sma200 AND  // Only look at stocks above 200 day moving average
  vol >= 300000 AND  // Filter out illiquid stocks
  Close >= 2.0;  // Filter out penny stocks
  
  
// ==========================================================================================
// === BUY & SELL LOGIC ===
// ==========================================================================================
// === NORGATE VARIABLES TO AVOID DESLISTED STOCKS ===
OnSecondLastBarOfDelistedSecurity = !IsNull(GetFnData("DelistingDate")) AND (BarIndex() == (LastValue(BarIndex()) -1) OR DateTime() >= GetFnData("DelistingDate") ) ;
OnLastTwoBarsOfDelistedSecurity = !IsNull(GetFnData("DelistingDate")) AND (BarIndex() >= (LastValue(BarIndex()) -1) OR DateTime() >= GetFnData("DelistingDate") );

// Buy criteria includes not buying stocks before they are delisted and they must be in historical SPX or NDX watchlists
Buy=Cross(High, Ref(DonchianUpper, -1)) AND Ref(Filter, -1)
  AND NOT OnLastTwoBarsOfDelistedSecurity
  AND (NorgateIndexConstituentTimeSeries("$SPX") OR NorgateIndexConstituentTimeSeries("$NDX"));

BuyPrice=Max(Ref(DonchianUpper, -1), Open);

// == TRAILING STOP ===
// Initialize variables
Sell=0;
trailArray = Null;
trailStop = 0;

//Trailing stop logic
for( i = 0; i < BarCount; i++ )
{  
  if( trailstop == 0 AND Buy[ i ] ) 
  { 
	trailstop = BuyPrice[i] * (1- stopLoss/100);
  }
  else Buy[i] = 0; // remove excess buy signals - having this here also ensures max stop loss can be hit, was avoiding entries otherwise.
  //if( trailstop > 0 AND Low[ i ] < trailstop )
  if( trailstop > 0 AND Low[ i ] < trailstop AND Buy[i] != True)  // adding Buy check ensures you don't stop out on the breakout candle when the stop is within it
   {   
	 Sell[ i ] = 1; 
	 SellPrice[ i ] = Min( Open[ i ], trailstop );
	 trailstop = 0;
   }
   
  if( trailstop > 0 )
   {
	  trailStop = Max(DonchianLower[i], trailStop); 
	  trailARRAY[ i ] = trailstop;
   }
}

Plot(trailArray, "Trailing Stop Level", colorRed, styleThick);

// Overwrite buy and sells to ensure not buying deslisted securities
Sell = Sell OR OnSecondLastBarOfDelistedSecurity;

// Plot arrows indicating trades
PlotShapes(IIf(Buy, shapeUpTriangle, shapeNone),color = colorLime, 0, L, Offset=-50);
PlotShapes(IIf(Sell, shapeDownTriangle, shapeNone),color = colorRed, 0, H, Offset=-50);


// ==========================================================================================
// === POSITION SCORE FOR RANKING STOCKS ===
// ==========================================================================================
PositionScore = vol; // sort by most liquid stocks

If you are using 'Pad & Align' (which is recommended when doing a portfolio backtest) then these lines of codes are not doing what you want them to do. Because 'Pad & Align' will keep padding fresh bars after symbol is deleted.

(BarIndex() == (LastValue(BarIndex()) -1) // Not correct logic when 'Pad & Align' is checked
(BarIndex() >= (LastValue(BarIndex()) -1) // Not correct logic when 'Pad & Align' is checked

Also with 'DateTime()' when not comparing for equality or inequality correct use is to use 'DateTimeDiff'.

See DateTimeDiff.

This is what I use:

dt = DateTime();
bi = BarIndex();

delistedSecurity = Nz(DateTimeDiff(dt, GetFnData("DelistingDate")) >= 0);
barsBeforeDelisting = LastValue(ValueWhen(dt == GetFnData("DelistingDate"), bi, 1)) -bi;
7 Likes

Thanks @TrendSurfer! I never would have expected the official FAQ of Norgate would have reference code that doesn't work correctly for portfolio backtesting. I'm glad you mentioned that.

I think I am following what you are doing here with your modification to their code. I replace

OnSecondLastBarOfDelistedSecurity = !IsNull(GetFnData("DelistingDate")) AND (BarIndex() == (LastValue(BarIndex()) -1) OR DateTime() >= GetFnData("DelistingDate") ) ;
OnLastTwoBarsOfDelistedSecurity = !IsNull(GetFnData("DelistingDate")) AND (BarIndex() >= (LastValue(BarIndex()) -1) OR DateTime() >= GetFnData("DelistingDate") );

with

dt = DateTime();
bi = BarIndex();
delistedSecurity = Nz(DateTimeDiff(dt, GetFnData("DelistingDate")) >= 0);
barsBeforeDelisting = LastValue(ValueWhen(dt == GetFnData("DelistingDate"), bi, 1)) -bi;

and replace

AND NOT OnLastTwoBarsOfDelistedSecurity;

with

AND NOT delistedSecurity

and replace

Sell = Sell OR OnSecondLastBarOfDelistedSecurity;

with

Sell = Sell OR barsBeforeDelisting == 1;

Is that correct? Full updated code below:

/*
NORGATE TEST
This strategy is just a simple strategy to ensure that the proper hooks to Norgate historical data are correct.
*/

// This should be included at the top of the AFL
#include_once "Formulas\Norgate Data\Norgate Data Functions.afl"

// ==========================================================================================
// === TRADING SYSTEM PARAMETERS ===
// ==========================================================================================
SetOption("InitialEquity", 100000);
SetTradeDelays( 0, 0, 0, 0 ); // 1 - Delays the trades to the next day, 0 - Trades same day via stops
SetOption("AllowPositionShrinking", True);
SetOption("AccountMargin",100); // 100 = Cash account, 75 = Using half available margin
SetOption("MaxOpenPositions", 20);
RoundLotSize = 1;  // Disallow fractional share purchasing


// ==========================================================================================
// === RISK MANAGEMENT & POSITION SIZING ===
// ==========================================================================================
// Initial (MAX) stop loss
SetOption("ActivateStopsImmediately", False);
stopLoss = 7; // Change this to a small number to experience the issue with trade entries being ignored
ApplyStop(stopTypeLoss, stopModePercent, stopLoss, ExitAtStop = 1);

// Position size based on risk and max stop loss
PositionRisk = 1;  
PosSize = 100 * PositionRisk / stopLoss;
SetPositionSize(PosSize, spsPercentOfEquity);
         

// ==========================================================================================
// === PLOT CHART & INDICATORS ===
// ==========================================================================================
// Chart
SetChartOptions(0,chartShowArrows|chartShowDates);
_N(Title = StrFormat("{{NAME}} - " + FullName() + " | {{INTERVAL}} {{DATE}}: Open %g, Hi %g, Lo %g, Close %g (%.1f%%) {{VALUES}}", O, H, L, C, SelectedValue( ROC( C, 1 ) ) ));
Plot( C, "Close", ParamColor("Color", colorDefault ), styleNoTitle | ParamStyle("Style") | GetPriceStyle() ); 

// SMAs
sma20 = ma(Close, 20);
sma200 = ma(Close, 200);
Plot(sma20,"20MA", color = colorGrey50, styleNoTitle);
Plot(sma200,"150MA", color = colorWhite, styleNoTitle);

// Donchian Channel
pdsU=4;  // High channel lookback
pdsL=5;  // Low channel lookback

DonchianUpper =HHV(H,pdsU);
DonchianLower = LLV(L,pdsL);

Plot(DonchianUpper,"DU",color = colorBlueGrey,styleDashed);
Plot(DonchianLower,"DL",color = colorBlueGrey, styleDashed);

// ==========================================================================================
// === FILTERING AND SCREENING CRITERIA ===
// ==========================================================================================
// Set other non-chart visible variables
vol = MA(Volume, 20); 

// Determine filtering criteria
Filter = 
  Close > sma200 AND  // Only look at stocks above 200 day moving average
  vol >= 300000 AND  // Filter out illiquid stocks
  Close >= 2.0;  // Filter out penny stocks
  
  
// ==========================================================================================
// === BUY & SELL LOGIC ===
// ==========================================================================================
// === NORGATE VARIABLES TO AVOID DESLISTED STOCKS ===
dt = DateTime();
bi = BarIndex();
delistedSecurity = Nz(DateTimeDiff(dt, GetFnData("DelistingDate")) >= 0);
barsBeforeDelisting = LastValue(ValueWhen(dt == GetFnData("DelistingDate"), bi, 1)) -bi;

// Buy criteria includes not buying stocks before they are delisted and they must be in historical SPX or NDX watchlists
Buy=Cross(High, Ref(DonchianUpper, -1)) AND Ref(Filter, -1)
  AND NOT delistedSecurity
  AND (NorgateIndexConstituentTimeSeries("$SPX") OR NorgateIndexConstituentTimeSeries("$NDX"));

BuyPrice=Max(Ref(DonchianUpper, -1), Open);

// == TRAILING STOP ===
// Initialize variables
Sell=0;
trailArray = Null;
trailStop = 0;

//Trailing stop logic
for( i = 0; i < BarCount; i++ )
{  
  if( trailstop == 0 AND Buy[ i ] ) 
  { 
	trailstop = BuyPrice[i] * (1- stopLoss/100);
  }
  else Buy[i] = 0; // remove excess buy signals - having this here also ensures max stop loss can be hit, was avoiding entries otherwise.
  //if( trailstop > 0 AND Low[ i ] < trailstop )
  if( trailstop > 0 AND Low[ i ] < trailstop AND Buy[i] != True)  // adding Buy check ensures you don't stop out on the breakout candle when the stop is within it
   {   
	 Sell[ i ] = 1; 
	 SellPrice[ i ] = Min( Open[ i ], trailstop );
	 trailstop = 0;
   }
   
  if( trailstop > 0 )
   {
	  trailStop = Max(DonchianLower[i], trailStop); 
	  trailARRAY[ i ] = trailstop;
   }
}

Plot(trailArray, "Trailing Stop Level", colorRed, styleThick);

// Overwrite buy and sells to ensure not buying deslisted securities
Sell = Sell OR barsBeforeDelisting == 1;

// Plot arrows indicating trades
PlotShapes(IIf(Buy, shapeUpTriangle, shapeNone),color = colorLime, 0, L, Offset=-50);
PlotShapes(IIf(Sell, shapeDownTriangle, shapeNone),color = colorRed, 0, H, Offset=-50);


// ==========================================================================================
// === POSITION SCORE FOR RANKING STOCKS ===
// ==========================================================================================
PositionScore = vol; // sort by most liquid stocks



You still want to keep the same logic OnLastTwoBarsOfDelistedSecurity but just coded differently.

Oh I think I see now. Your two pieces of code you gave me above were the same two statements in the original Norgate variables broken out individually and re-worded.

So the two original Norgate variables could be rewritten as:

OnSecondLastBarOfDelistedSecurity = barsBeforeDelisting == 1 OR delistedSecurity;
OnLastTwoBarsOfDelistedSecurity = barsBeforeDelisting == 2 OR delistedSecurity;

And then just use those variables as I had them originally in the Buy/Sell logic. Is that right?

Use like this:

OnSecondLastBarOfDelistedSecurity = barsBeforeDelisting == 1; // Incorporate into your Sell rules (Sell if true)
OnLastTwoBarsOfDelistedSecurity = barsBeforeDelisting == 2 OR barsBeforeDelisting == 1; // Incorporate into your Buy rules (Don't Buy if true)
2 Likes

Would you also include the delistedSecurity variable (as shown below)? If not, where is that used/why did you add that to your original answer? Seems like it would be potentially useful to have as a catchall for not grabbing a security that is known to be delisted, but maybe it's handled entirely within the barsBeforeDelisting check.

OnSecondLastBarOfDelistedSecurity = barsBeforeDelisting == 1 OR delistedSecurity; // Incorporate into your Sell rules (Sell if true)
OnLastTwoBarsOfDelistedSecurity = barsBeforeDelisting == 2 OR barsBeforeDelisting == 1 OR delistedSecurity; // Incorporate into your Buy rules (Don't Buy if true)

Yep exactly, Filter = NOT delistedSecurity.

2 Likes

This topic was automatically closed 100 days after the last reply. New replies are no longer allowed.