Getting null values for StaticVarGenerateRanks

AFL Programming
1

I'm sure I'm doing something stupid but I'm not seeing it.

Anyone know why my SVGR ranks are all null?

Also, I would have expected turnover (StaticVarGet) and check_turnover (derived again in "open" code) to be equal?

Lastly, does my calculation for average total turnover look correct? I can export data to Excel and check my math - especially null handling - but if you see something obvious please let me know.

// One-off loop
if (Status("stocknum") == 0)
{
	// Get list of symbols from current watchlist
	symbollist = CategoryGetSymbols(categoryWatchlist, GetOption("FilterIncludeWatchlist"));

    // cleanup variables created in previous runs (if any)
    StaticVarRemove("*");

	// derive our desired metric for each symbol in the watchlist
	sumturnover=0;
	cntturnover=0;
    for (i=0; (symbol = StrExtract(symbollist,i)) != ""; i++)
    {
		// set the OHLC arrays to the current symbol
        SetForeign(symbol);
			// derive our desired metric(s) for each symbol in the watchlist
			ibdrs = 0.4 * ROC(C,63) + 0.2 * ROC(C,126) + 0.2 * ROC(C,189) + 0.2 * ROC(C,252);
			turnover = MA(C*V,20);
			
			// say we also want to derive the average turnover for all the stocks
			sumturnover+=Nz(turnover);				// if the array element is null it is converted to zero
			cntturnover+=NOT(IsNull(turnover));		// if the array element is null it equals 1 else 0, NOT reverses the number
        RestorePriceArrays();

        // save the metric to a static variable
        // the static varname must be unique per symbol, i.e. include "symbol" in its name
        StaticVarSet("ibdrs" + symbol, ibdrs);
        StaticVarSet("turnover" + symbol, turnover);
    }
    
    // save the average turnover for all the stocks to a static variable
    avgturnover=SafeDivide(sumturnover,cntturnover);
    StaticVarSet("avgturnover",avgturnover);

    // now that we have a time series array for each symbol,
    // use StaticVarGenerateRanks to generate the rank for each symbol
    
    // I believe the way this works is StaticVarGenerateRanks reads all the static vars with that prefix name,
    // sorts them, then generates a rank number and saves that to the output prefix + symbol 
    StaticVarGenerateRanks("rank_ibdrs","ibdrs",0,1224);
    StaticVarGenerateRanks("rank_turnover","turnover",0,1224);
 }

// Now that we are in open code, the first pass of the backtest engine loops over every symbol in the watchlist
symbol = Name();

ibdrs 			= StaticVarGet("ibdrs" + symbol);
check_ibdrs		= 0.4 * ROC(C,63) + 0.2 * ROC(C,126) + 0.2 * ROC(C,189) + 0.2 * ROC(C,252);
rank_ibdrs		= StaticVarGet("rank_ibdrs" + symbol);

turnover		= StaticVarGet("turnover" + symbol);
check_turnover	= MA(C*V,20);
rank_turnover	= StaticVarGet("rank_turnover" + symbol);

avgturnover 	= StaticVarGet("avgturnover");

// pretend buy signals
Buy = rank_ibdrs < 3
  AND rank_turnover < 10
  AND turnover > avgturnover
;

// exploration code for debugging
Filter = 1;
//Filter = Buy;

AddColumn(ibdrs,"ibdrs");
AddColumn(check_ibdrs,"check_ibdrs");
AddColumn(rank_ibdrs,"rank_ibdrs");
AddColumn(turnover,"turnover");
AddColumn(check_turnover,"check_turnover");
AddColumn(rank_turnover,"rank_turnover");
AddColumn(avgturnover,"avgturnover");  // should be the same for all symbols for a given date

if (Status("Action") == actionExplore) SetSortColumns(2,5,8);
2

Did you run your code under debugger so you can actually WATCH the variables?

To get better understanding of what is happening in your code and how functions work, use advice given here: How do I debug my formula?

3

Yes I did. I'm quite familiar with using the debugger from my role as a professional computer programmer.

At least for the StaticVarGenerateRank question, please enlighten me how I can WATCH its execution under the debugger?

4

Read the documentation for SVGR again. You're using the wrong variable name for StaticVarGet when trying to retrieve the rankings.

2 Likes
5

Thanks Matt. Like I said I knew it was something stupid.

Even with your reply and my re-reading the SVGR doc, I still wasn't seeing it. Sometimes the mind just isn't working properly. But your reply did trigger an idea I hadn't though of before I posted: StaticVarInfo, which proved to be a useful debugging function.

Here is my revised code in case it might help someone who finds this post in the future.

Now to work out why turnover != checkturnover for many rows??? AFAIK they should be the same, as they are for ibdrs.

// One-off loop
if (Status("stocknum") == 0)
{
	// Get list of symbols from current watchlist
	symbollist = CategoryGetSymbols(categoryWatchlist, GetOption("FilterIncludeWatchlist"));

    // cleanup variables created in previous runs (if any)
    StaticVarRemove("*");

	// derive our desired metric for each symbol in the watchlist
	sumturnover=0;
	cntturnover=0;
    for (i=0; (symbol = StrExtract(symbollist,i)) != ""; i++)
    {
		// set the OHLC arrays to the current symbol
        SetForeign(symbol);
			// derive our desired metric(s) for each symbol in the watchlist
			ibdrs = 0.4 * ROC(C,63) + 0.2 * ROC(C,126) + 0.2 * ROC(C,189) + 0.2 * ROC(C,252);
			turnover = MA(C*V,20);
			
			// say we also want to derive the average turnover for all the stocks
			sumturnover+=Nz(turnover);				// if the array element is null it is converted to zero
			cntturnover+=NOT(IsNull(turnover));		// if the array element is null it equals 1 else 0, NOT reverses the number
        RestorePriceArrays();

        // save the metric to a static variable
        // the static varname must be unique per symbol, i.e. include "symbol" in its name
        StaticVarSet("ibdrs" + symbol, ibdrs);
        StaticVarSet("turnover" + symbol, turnover);
    }
    
    // save the average turnover for all the stocks to a static variable
    avgturnover=SafeDivide(sumturnover,cntturnover);
    StaticVarSet("avgturnover",avgturnover);

    // now that we have a time series array for each symbol,
    // use StaticVarGenerateRanks to generate the rank for each symbol
    
    // I believe the way this works is StaticVarGenerateRanks reads all the static vars with that prefix name,
    // sorts them, then generates a rank number and saves that to the output prefix + symbol 
    StaticVarGenerateRanks("rank","ibdrs",0,1224);
    StaticVarGenerateRanks("rank","turnover",0,1224);
 }

// Now that we are in open code, the first pass of the backtest engine loops over every symbol in the watchlist
symbol = Name();

// Useful for debugging, run this in the debugger
printf("StaticVarList: %s",StaticVarInfo("*","list"));

ibdrs 			= StaticVarGet("ibdrs" + symbol);
checkibdrs		= 0.4 * ROC(C,63) + 0.2 * ROC(C,126) + 0.2 * ROC(C,189) + 0.2 * ROC(C,252);
rankibdrs		= StaticVarGet("rankibdrs" + symbol);

turnover		= StaticVarGet("turnover" + symbol);
checkturnover	= MA(C*V,20);
rankturnover	= StaticVarGet("rankturnover" + symbol);

avgturnover 	= StaticVarGet("avgturnover");

flag1			= Nz(ibdrs) == Nz(checkibdrs);
flag2			= Nz(turnover) == Nz(checkturnover);

// pretend buy signals
Buy = rankibdrs <= 3
  AND rankturnover <= 10
  AND turnover > avgturnover * 1.2  // this stock's turnover is greater than 1.5 times the average turnover of all the stocks
;

// exploration code for debugging
Filter = 1;
//Filter = Buy;

AddColumn(ibdrs,"ibdrs");
AddColumn(checkibdrs,"checkibdrs");
AddColumn(rankibdrs,"rankibdrs",1.0);
AddColumn(turnover,"turnover");
AddColumn(checkturnover,"checkturnover");
AddColumn(rankturnover,"rankturnover",1.0);
AddColumn(avgturnover,"avgturnover");  // should be the same for all symbols for a given date
AddColumn(flag1,"ibdrs_flag",1.0);
AddColumn(flag2,"turnover_flag",1.0);

if (Status("Action") == actionExplore) SetSortColumns(2,5,8);
6

One hint that you might not be aware of.
Static variables can be displayed in the debugger directly.
Just append '$' (dollar) prefix before static variable name in the WATCH window.

https://www.amibroker.com/guide/h_debugger.html

As to your turnover question - your code is using FIRST symbol as reference. If that symbol happens to have less quotes than other symbols (or other holes/extra datas) the Foreign would PAD AND ALIGN to that symbol.

This is described in the manual
https://www.amibroker.com/f?foreign

Obviously this changes data as compared to NON-padded data.

I recommended several times AGAINST using loop in case of StockNum == 0.

Instead use #pragma sequence and create static variables during SCAN, and LATER use StaticVarGenerateRanks in second sequence step.

2 Likes
7

Thanks Tomasz, I was not aware of this hint.

While I have read the Amibroker PDF cover to cover, it is difficult to memorize it in its entirety. So apologies if I post the occasional question asking for assistance. I've re-read the debugger link above and TBH didn't (re)learn anything I didn't already know. There is also no mention of the Static variables hint. If it is documented elsewhere please send me the link, otherwise thanks for the hint. Perhaps the debugger link you've provided can be updated to include that hint?

Sorry I'm not following this. Are you referring to my derivation of total average turnover or the Amibroker formulas for the individual symbol turnover? My question was referring to the latter.

I would expect:

turnover = MA(C*V,20);  StaticVarSet("turnover" + symbol, turnover);

derived in the stocknum == 0 loop...and

checkturnover	= MA(C*V,20);

derived in the first pass of the backtest engine to be equal for the same symbol.

I generally run with Norgate padding All Market Days and no Analysis settings padding. I also tried all permutations of Norgate padding plus P&A in Analysis settings, all with differing results for the two turnover formulas. The ibdrs derivation is identical for both the StaticVar and backtest engine derivation.

I based my code on this knowledgebase article - obviously modified but using the same general code structure:

(I'm getting an error "Sorry you cannot post a link to that host", trying to post a link to the Amiborker knowledgebase!!! Go figure?! Anyway, it's the KB article titled "Separate ranks for categories that can be used in backtesting")

Is this KB article obsolete? If so perhaps it can be updated with a link to your suggested approach?

Other than efficiency (yes I see the Warning 512 in the formula window) should that approach still work? Regardless, I'm still not following why the two ibdrs derivations match and the turnover derivations don't.

8
  1. It is documented only in the Release Notes and it "What's new" in the Users' Guide.

CHANGES FOR VERSION 6.19.0 (as compared to 6.18.0)
Debugger: Watch window: added ability to display values of static variables in watch window - just append '$' (dollar) prefix before static variable name

and in the announcements for version 6.19 and 6.20 AmiBroker DevLog » AmiBroker 6.20.1 Release Candidate

When new features are introduced they are always described in detail in announcements and in Release Notes. Summary of changes is included in Whats New page in the manual. When installing new versions it is good idea to check this as it makes it fastest way to get familiar with new features without need to look in entire manual.

  1. You are using SetForeign in you loop. SetForeign CHANGES the symbol from first one to the one that you selected but it also performs padding to selected (in that case first) symbol as documented in AFL Function Reference - FOREIGN (read the comment part)
    So your static vars contain data padded to FIRST symbol, while the other statement not. As to Norgate: I can't speak about how Norgate is padding data since I did not write that and I am not using Norgate data.
    I provided you with with a "guestimate" of what could be the reason. I did not run your code and did not spend hours on it, since I can't do that (I already struggle to do my own stuff).
    As I wrote use How do I debug my formula? as tool to find the reason of what you see. For example add _TRACE() to find out what BARCOUNT is for first symbol and what BARCOUNT is for the symbol you have trouble with. Different BarCounts would give you a hint that your data are not equal length and subject to padding. Also use Tools->Database Purify to find out errors in database.

  2. The software evolves. KB article was written BEFORE #pragma sequence was introduced. The approach described in KB still works, but it is subject to padding.

  3. As to error message from forum (I guess), please send a link you are trying to post and screenshot of error message to support address (email). Generally posting links is allowed, but Discourse may choke if too many links are copy-pasted as it starts to think it is "spam".

1 Like
9

This topic was automatically closed 100 days after the last reply. New replies are no longer allowed.