Exploration ranking using multiple watchlits

millerrh · August 19, 2023, 7:22pm

I am interested in performing a ranking based on relative strength as a high level screener and taking the top X performers. Then use those as my list of tickers to perform subsequent analysis on.

I found this example code below from this post which is very similar, to what I was thinking of.

However, GetOption( "FilterIncludeWatchlist" ); will only take a single watchlist in the Filter settings. I was actually interested in using multiple watchlists as my input. I thought maybe using the watchlist names and adding the resulting symbol Lists together would be a way around that. I attempted to do that with one watch list name and it does not seem to work correctly.

Any help on how to run a ranking on multiple watchlits would be appreciated.

// ################ Ranking Exploration #################

// watchlist should contain all symbols included in the test
//wlnum = GetOption( "FilterIncludeWatchlist" );
wlname = "S&P 500";
wlnum = CategoryFind( wlname, categoryWatchlist );
List = CategoryGetSymbols( categoryWatchlist, wlnum ) ;

if( Status( "stocknum" ) == 0 )
{
    // cleanup variables created in previous runs (if any)
    StaticVarRemove( "rank*" );
    StaticVarRemove( "values*" );
    categoryList = ",";

    for( n = 0; ( Symbol = StrExtract( List, n ) )  != "";  n++ )
    {
        SetForeign( symbol );

        // write our ranking criteria to a variable
        // in this example we will use 10-bar rate-of-change
       values = ROC(C,10);

        RestorePriceArrays();

        // write ranked values to a static variable
        StaticVarSet( "values_" + symbol, values );

    }

    // generate  ranks
    StaticVarGenerateRanks( "rank", "values_", 0, 1224 );
    
}

symbol = Name();
values = StaticVarGet( "values_" + symbol );
rank = StaticVarGet( "rankvalues_" + symbol );

// exploration code for verification
AddColumn( values, "ROC" );
AddColumn( rank, "Rank" );
Filter = rank <= 6;

Tomasz · August 20, 2023, 12:27pm

You can use just use
InWatchListName() function
without any filtering at all in the analysis window.

millerrh · August 20, 2023, 1:43pm

Hi @Tomasz,

That doesn't exactly give me what I'm looking for as InWatchListName() gives a true/false value. I actually do use this function already in my Interpretation window for checking if a given ticker is within a watchlist.

What I am looking to do is the following, perhaps explaining it better in English would be better.

Filter a large universe of stocks from multiple watchlists down to a smaller universe (like top 200) based on market strength, like ROC.
Take that single new combined list and perform other filtering criteria to it that meet the specific criteria of my strategy, like minimum volume, dollar value, ADR, etc.
Export that final filter as an Exploration that can be imported into a new "nightly screen" watchlist and/or be copied as an Excel table.

The first step of coming up with a single ranked list based on multiple watchlists is something I am not clear on.

beppe · August 20, 2023, 5:59pm

I think @Tomasz suggsted to explore/scan selecting "*All symbols" in the filter dropdown, and then use multiple OR InWatchListName() in your ranking step code (with each call checking a specific watchlist).

As an alternative I suggest to avoid the use of "Foreign" and replace that code with the one in this thread selecting multiple watchlists in the filter dialog:

And you can probably consolidate points 1 and 2 into a single step by simply assigning a "low" value to stocks that don't match the criteria in point 2.

millerrh · August 20, 2023, 8:12pm

Thank you @beppe for the follow up. I clearly misunderstood what @Tomasz was referencing in his reply and I think this may very well be the solution I am looking for indeed. Learned more about #pragma sequence( scan, explore ) in this post and this help article.

I tried to adapt the non-Foreign script you referenced with the ROC function for "values" and display the Filter information for the Exploration. Clicking on the "Run Sequence" button fails to display anything in the Analysis Window though. Despite reading about the new #pragma sequence I'm likely still not completely understanding how the structure should be.

Here is my code:

#pragma sequence( scan, explore )

if( Status( "action" ) == actionScan )
{
    if( Status( "StockNum" ) == 0 )
    {
        StaticVarRemove( "ValuesToSort*" );
    }

	Values = ROC(C,10); // NO NEED TO CALL Foreign !

    StaticVarSet( "ValuesToSort" + Name(), values );

	_exit(); // we are done - don't proceed further
}

if( Status("StockNum") == 0 )
{
   //remove previous rankings if any
  StaticVarRemove( "rank*" );
   //do the ranking 
  StaticVarGenerateRanks("rank", "ValuesToSort", 0, 1224);
}

symbol = Name();
values = StaticVarGet( "ValuesToSort" + symbol );
rank = StaticVarGet( "rank" + symbol );

// exploration code for verification
AddColumn( values, "ROC" );
AddColumn( rank, "Rank" );
Filter = rank <= 6;

beppe · August 21, 2023, 7:45am

@millerrh, this line is wrong:

rank = StaticVarGet( "rank" + symbol );

please, replace it with:

rank = StaticVarGet("rankValuesToSort" + symbol);

millerrh · August 21, 2023, 8:16pm

@beppe, this now does perform an action, but it is not ranking the tickers as I was wanting, which was why I was originally defining the rank variable as the "rank" Static Variable.

rank = StaticVarGet( "rank" + symbol );

My hope with this code was that I could associate the rank value of each ticker and then limit the number of items that show up to the top X ranked items.

Now in the exploration, the columns list ROC and Rank and they are one and the same and I'm missing that ranking. Here is a sample of the results where the Rank is simply being assigned the ROC value per your recommendation.

beppe · August 22, 2023, 9:28am

@millerrh, to get the rank values you should use the correct StaticVar name:

From the documentation:

To fill input static variables you can use this loop:

for ( i = 0; ( sym = StrExtract( symlist, i ) ) != ""; i++ )
{
    SetForeign( sym );
    Value = ROC( C, 10 );
    RestorePriceArrays();
    StaticVarSet( "ItemScore" + sym, Value );
}

Now you are ready to perform sorting/ranking. ...

StaticVarGenerateRanks( "rank", "ItemScore", 0, 1224 );

In this case StaticVarGenerateRanks call would generate set of static variables starting with prefix defined by 2nd argument each variable holding the rank of particular symbol, so in this case RankItemScoreMSFT will hold ranking of MSFT, RankItemScoreAAPL will hold ranking of AAPL. Note that in AmiBroker rank count start from ONE.

The key point is that the name of the static variable that stores the rank is created using - as a prefix - the string concatenation of the 1st and the 2nd parameter:

StaticVarGenerateRanks( "rank", "ItemScore", 0, 1224 );

so the prefix should be: rankItemScore (not just "rank" as you did).

Below is essentially the code you posted with the fix I suggested:

#pragma sequence( scan, explore )

if( Status( "action" ) == actionScan )
{
    if( Status( "StockNum" ) == 0 )
    {
        StaticVarRemove( "ValuesToSort*" );
    }

	Values = ROC(C,10); // NO NEED TO CALL Foreign !

    StaticVarSet( "ValuesToSort" + Name(), values );

	_exit(); // we are done - don't proceed further
}

if( Status("StockNum") == 0 )
{
   //remove previous rankings if any
  StaticVarRemove( "rank*" );
   //do the ranking 
  StaticVarGenerateRanks("rank", "ValuesToSort", 0, 1224);
}

symbol = Name();
values = StaticVarGet( "ValuesToSort" + symbol );
rank = StaticVarGet( "rankValuesToSort" + symbol ); // This was the wrong line in the OP code

// exploration code for verification
if( Status( "action" ) == actionExplore )
{
	AddColumn( values, "ROC" );
	AddColumn( rank, "Rank", 1 );
	Filter = rank <= 6;
	SetSortColumns(-2, 4);
}

And it works as expected:

Finally, when doing ranking the recommendation is to enable "Pad and align" in the "Analysis settings" dialog. Be sure to use as reference a symbol that has a complete and reliable history (no data holes).

Also, the idea of "exporting" the best stocks to a watchlist can only be useful if you take into consideration a single bar (e.g. like in your screenshot, the last one to create a watchlist of stocks to be "monitored" live at the next opening).
The code, if applied to a historical series, must "virtually create" a universe on which to operate by assigning a low ranking value to stocks that do not meet certain criteria which automatically excludes them from any "buy" signals.

beppe · August 22, 2023, 10:14am

@millerrh, btw, if your goal is just to create a daily watchlist, with Amibroker version 6.43, probably you can simplify the process using a formula similar to this code (running an exploration):

Version( 6.43 );
conditionA = true; // change with criteria to include stocks only from some watchlists - InWatchlistName("abc") OR InWatchlistName("yyz") ...
conditionB = true; // change with criteria to exclude some stocks (volume, min price, etc.)
Filter = conditionA AND conditionB;
AddColumn( ROC( C, 10 ), "ROC" );
SetSortColumns( -3 );
AddRankColumn(); // optional - but visually nice
TrimResultRows( 6 ); // trims the result list to specified number of rows, positive row count means counting from top, negative from bottom.  Requires version >= 6.43.1

millerrh · August 22, 2023, 9:53pm

Thanks @beppe, the concatenation of the 1st and the 2nd parameters is something I didn't figure out when looking at the documentation earlier somehow. This now works like I was hoping.

As to your other comments, when I screen each night using the exploration, I do only use the previous bar (as shown in the screen shot).

The way I have done it is I have my explorations for each strategy embedded in the strategy code and have my buy signals only trigger if the previous day's Filter is true, so that should take care of applying it to a historical series. I have been doing it this way for some time and my explorations match my buys and sells in real time.

The one thing that I was hoping to do with this exercise is limit my universe of stocks before I apply my typical filters to them. In my current Filter criteria, I have a min rate of change, but I'd like my rate of change filter to fluctuate with the market health, so that I'm always looking at the top X number or X% of stocks as my main universe before applying other criteria. Understanding how static variables work and to rank them (and count the total so I can take the % of stocks) is the first step in doing this.

I really appreciate all the help you've given to me in this thread. While you technically solved my question with your last reply, I think I will keep this open in case a further question arises as I expand on this exploration idea. I'll definitely mark as solved later if I have no further questions.

millerrh · August 22, 2023, 10:17pm

Actually as I play with this more, I wonder if the #pragma sequence function is what I want if my end goal is to embed the Exploration Filtering within a strategy so that I can check if the filter was true the previous day.

It does give me the ability to look at multiple watchlists and condense them into a single list of ranked tickers like I was looking for, but ultimately I also want that same functionality built into a system.

Maybe I need to go back to the Foreign method instead? If so is there a way to call multiple watchlists?

beppe · August 23, 2023, 6:47pm

@millerrh, whether to use the loop with "Foreign()" or the scan/explore sequence is a choice you are free to make (although using multiple Foreign calls is now discouraged).

Regardless of the approach used, you'll get the exactly the same results in terms of ranking.

What changes is the execution speed of the formula.
The "Foreign" method is significantly slower (especially if you have to check many hundreds of stocks) since it needs to be executed in a single thread; search the forum for multiple examples/discussions about it.

The "sequence" is simply a convenience to "automate" a common scenario: doing a scan followed by an exploration.

You can do the same manually by running a "scan" (at full speed since it is multi-threaded) that processes all the stocks included in the Analyisis filter dialog creating the staticVars of "values" to be ranked.
Then you can run the exploration that actually "calculates" the ranking retrieving the "staticVars" created by the scan.

Nothing prevents you from adding in the same formula the necessary logic to "buy/sell" some titles based on the resulting ranking.

If you still prefer the "Foreign" way, reread the @Tomasz suggestion if you have many watchlists to process, or use the Analysis "filter" dialog to select up to seven watchlists. In the latter case, in your code before the "Foreign" loop you should retrieve and combine all the symbols from each of the selected watchlists to a single huge list of comma-separated tickers to be processed.

millerrh · August 23, 2023, 8:02pm

Thanks @beppe. Since I want to include this in the strategy code itself eventually, it seems like I can't use the #pragma sequence function as I don't have access to the "Run Sequence" button within a strategy. Is that correct? That's why I was assuming I'd need to go back to using the Foreign calls.

As for the ability to call multiple watchlists manually as per my original question, I think I figured it out. The key (but obvious) thing is that all your watchlists are also selected in the Filter window if using the "Apply to: *Filter" selection of the exploration. Once that is done, I was able to search through multiple lists, add them together, and come up with a single ranked universe.

// ################ Ranking Exploration #################
// Make sure all watchlists you are exploring are selected in the Filter window for Explorations!
wlname1 = "S&P 500";
wlnum1 = CategoryFind( wlname1, categoryWatchlist );
wlname2 = "Nasdaq 100";
wlnum2 = CategoryFind( wlname2, categoryWatchlist );
List1 = CategoryGetSymbols( categoryWatchlist, wlnum1 ) ;
List2 = CategoryGetSymbols( categoryWatchlist, wlnum2 ) ;
List = List1 + "," + List2;

if( Status( "stocknum" ) == 0 )
{
     //cleanup variables created in previous runs (if any)
    StaticVarRemove( "rank*" );
    StaticVarRemove( "values*" );
    //categoryList = ",";

    for( n = 0; ( Symbol = StrExtract( List, n ) )  != "";  n++ )
    {
        SetForeign( symbol );

         //write our ranking criteria to a variable in this example we will use 10-bar rate-of-change
       values = ROC(C,10);

        RestorePriceArrays();

         //write ranked values to a static variable
        StaticVarSet( "values_" + symbol, values );

    }

     //generate  ranks
    StaticVarGenerateRanks( "rank", "values_", 0, 1224 );
    
}

symbol = Name();
values = StaticVarGet( "values_" + symbol );
rank = StaticVarGet( "rankvalues_" + symbol );

 //exploration code for verification
AddColumn( values, "ROC" );
AddColumn( rank, "Rank" );
Filter = rank <= 6;
SetSortColumns(-2, 4);

Tomasz · August 24, 2023, 7:04am

For what it is worth, I was suggesting using InWatchListName() function to perform filtering entirely inside formula ("Apply to" would then be "All symbols").
Filter dialog as @beppe wrote provides alternative way of selecting multiple watchlists without need to code anything.

millerrh · August 24, 2023, 1:22pm

Thanks @Tomasz, I also tried that but didn't like how long it takes to search through all the symbols.

My end goal is to embed the exploration code within a strategy, so that my Buy condition can have the condition of Ref(Filter, -1) being True added to it so that I am able to backtest how my nightly screen exploration performs over time.

In this use case I think I do need the Static Variable and it being coded up so that I can refer to it historically. Please correct me if I'm wrong about that assumption.