Premium Data - Help Speeding Up Loop/Threading

I have been using premium data for several years. I have one strategy where I have to loop through all constituaents of the S&P 500 Current and Past. My backtest is extremely slow.... can anyone look at the code and see if there is a way to speed this up.... Here is what the code does

  1. Backtest back to 1991

  2. If the stock is in S&P500 at the time IIf(NorgateIndexConstituentTimeSeries("$SPX")

  3. Set Yield Variable... Set DivYears (# of years of constant or increasing divs)

  4. I then rank on yield variable

Finally I have filters and buy logic to buy on volatility..... my problem is the slow speed it loops through the watchlist.... im hoping there is a way to speed this up... thanks.. here is my code for the watchlist loop....

if ( GetOption( "ApplyTo" ) == 2 ) 
      wlnum = GetOption( "FilterIncludeWatchlist" ); 
      List = CategoryGetSymbols( categoryWatchlist, Wlist ) ; //************S&P 500 HISTORICAL CONST CONFIRM WATCHLIST **************************
if ( GetOption( "ApplyTo" ) == 0 ) 
      List = CategoryGetSymbols( categoryAll, 0 ); 
      //Error( "The formula works fine if your ApplyTo setting is 'Filter' or 'All' " ); 

if ( Status("stocknum") == 0 ) // GENERATE RANKING WHEN WE ARE ON VERY FIRST SYMBOL 
     StaticVarRemove( "values*" ); 
     StaticVarRemove( "DivYrs1*" ); 
     for ( n = 0; ( Symbol = StrExtract( List, n ) )  != "";  n++    ) 
        SetForeign ( symbol );
			ODIV = NorgateAllDividendsTimeSeries(1);	
			Bars0a = BarsSince(Odiv);
			Bars1a = Ref(bars0a,-1);
			B1a = bars1a+2;
			Bars2a = Ref(bars0a,-b1a);
			B2a = B1a+bars2a+1;
			Bars3a = Ref(bars0a,-b2a);
			B3a = B2a + bars3a+1;
			Bars4a = Ref(bars0a,-b3a);
			Bars12a = bars1a+bars2a+bars3a+bars4a;
			Div252a = Sum(ODIV,bars12a);
			AnnualDividend = IIf(ODIV>0,div252a,Ref(div252a,-Bars0a));			
			DivRise = AnnualDividend>0 AND AnnualDividend>= Ref(AnnualDividend*.99,-bars1a);//Dividend Increase Years//////////////////////////////////////////
			DivFall = AnnualDividend<Ref(AnnualDividend*.99,-bars1a);
			DivRise = ExRem(DivRise,Divfall);
			DivFall = exrem(DivFall,DivRise);
			DivIncreaseYrs = BarsSince(DivRise)/250;
			values = IIf(NorgateIndexConstituentTimeSeries("$SPX"),yield,-100);
			DivYrs1 = IIf(NorgateIndexConstituentTimeSeries("$SPX"),DivIncreaseYrs,0);
        StaticVarSet (  "values"  +  symbol, values ); 
        StaticVarSet (  "DivYrs1" + Name() , DivYrs1 ); 
        _TRACE( symbol ); 
     StaticVarGenerateRanks( "rank", "values", 0, 1224 ); 

symbol = Name(); 
values = StaticVarGet ( "values" +  symbol ); 
rank = StaticVarGet ( "rankvalues" +  symbol );  

Capitalization = GetFnData("SharesOut")*Close;
DailyDollar = MA(Volume*Close,252);
VolFilter1 = Capitalization < 500000000;
dollarFilter = dailydollar < 250000;
VolFilter = Volfilter1 OR dollarfilter;// 

RankFilter = rank > 100;
Volatility =IIf(volfilter,100,IIf(rankfilter,100,StDev((C-Ref(C,-1))/Ref(C,-1),Volper)*100)); 

VolAvg = IIf(NorgateIndexConstituentTimeSeries("$SPX"),Volatility,100);
VolAvg = IIf(IsFinite(Volavg),Volavg,100); 

ScoreAvg = WMA(volavg,20);
Score= 100-scoreavg;
PositionSize = -100/Totalpositions;  
PositionScore = Score;
PositionScore = IIf(PositionScore < 0, 0, PositionScore);
PositionScore = IIf(Rebalance, PositionScore, scoreNoRotate);
Filter = Rebalance;

@mfu5324 I have not looked over your code because I think you are mistaken about what your data vendor is providing you. To the best of my knowledge, Norgate provides current fundamental data, not historical fundamental data. So you can not calculate rising or falling dividends for a stock because you do not have that data.

Read their web page,


Actually I am getting the didvdends based on this series which does go back. I am creating my own yi or calf

ODIV = NorgateAllDividendsTimeSeries(1);

@mfu5324 Ok, nice idea while we wait for @NorgateData to finish development on

under active development as of July 2018 is an indicator called NorgateDividendYieldTimeSeries that will show a trailing time series of a trailing 12 month dividend yield for all the history of that stock, allowing you to backtest a time series calculated for each historical bar showing you the dividend yield of the stock based upon the current dividend entitlements for that stock at that particular date.

I tried running your code but I get this error (I too have same data),

The other suggestion is to look into using SparseCompress since your data series has many zero's that you don't need.

@mfu5324 I was timed out as I was editing my previous post, but something like this (very preliminary) may help?

#include_once "Formulas\Norgate Data\Norgate Data"

ODIV = NorgateAllDividendsTimeSeries( 1 );

x = SparseCompress( ODIV != 0, ODIV );
FourQuarters = Sum( x, 4 );
y = SparseExpand( ODIV != 0, Fourquarters );

Filter = odiv != 0;
AddColumn( odiv, "odiv" );
AddColumn( y, "Sum of Last 4 dividends" );


Well that is definatly a better way of getting th edividend than what i was doing.....sparse.... unfortunately this didnt solve my issue.... Norgate has historical constuens back to 1950's and this is why my code is so slow.... alot of symbols in that watchlist..

My solution is to exclude all symbols prior to 1991 (when i started backtest) AND i am going to exclude all delisted symbols that i know are not part of the strategy... i just need a little help

  1. Run explore on S&P Current and Past... and create new watchlist and DO NOT add symbols delisted prior to 1991... i can do this

  2. I then took the remaining delisted stocks that are not part of the strategy and saved them to a csv.

  3. My problem is, im trying to read the csv and for each symbol in the csv, remove them from my new watchlist using CategoryRemoveSymbol.... but i cant get that to removes them all or none....

***I HOPE I CAN DO THIS WITHOUT LOOPING THROUGH THE WATCHLIST... rather doa simple exploration and for each symbol in csv, remove from the new watchlist... any help?

Watchlist = "S&P-1991";
Wlist = CategoryFind(Watchlist,categoryWatchlist);
//wlnum = GetOption( "FilterIncludeWatchlist" ); 

StaticVarRemove( "SYM_*" );
ct = 0;
EnableTextOutput( 3 ); 
// get symbols in csv that i want to remove from my new wathclist
fh = fopen( "//{1B9V7A2I}//DIVGExclude2.csv", "r"); 					
		while( ! feof( fh ) )
		 ct = ct+1;
			 line = fgets( fh );	 
			 symX = StrExtract( line, 0 );
		 fclose( fh );
		 Error( "ERROR: file can not be open" );
/// here is where i filter out all delisted symbols prior to 1991
sym = Name();
YR = StrRight(sym,6);
YR2 = StrToNum(StrLeft(YR,4));
YR3 = YR2==0 OR YR2>1990;
Filter=1 AND YR3;
///////////////////////////////////////////////////now add symbols to S&P-1991 watchlist
CategoryAddSymbol("",categoryWatchlist, Wlist ) ;

// here is where i try to remove them from the new wathclist... but its removing incorrect ones

for( j = 1; j < ct ; j++ )
    Sym_ = staticVarGettext("SYM_" + j);
    CategoryRemoveSymbol(Sym_,categoryWatchlist, Wlist ) ;

Is this step something that could be performed with AFL? If so, then just make that criteria part of the Exploration that you're doing in Step 1 so that the result of the Exploration is ONLY those symbols that you want to run your backtest against.

The issue with dividend yields is that the ex-dates for each dividend can vary a little from year-to-year.

Also, dividend yields are only to be calculated on capital event+special dividend adjusted data (so that special dividends, spinoffs etc. don't distort the dividend yield).

There are also other considerations, such as dividends occuring before your chart plot, which you will likely want to incorporate.

We're putting the final touches on our dividend yield indicator, which will be available through NorgateDividendYieldTimeSeries(). Here's a sneak-peek:



(right click each image, open image in new tab for a detailed picture)

Thanks for the update. That looks like really good feature your adding.

The NorgateDividendYieldTimeSeries() indicator has been released. Usage details are here: