Help with AddToComposite()


#1

I’m trying with the following code to create an index of S&P stocks called “~bob_all_stocks” with ROC values in the Close field and the total number of contributing stocks in the Volume field. The image below the code shows my results.
I’d be grateful for advice on why I’m getting all sorts of exploration stock counts rather than the approximately 500 I was expecting for each row. Thanks very much.


wlnum = 985; //S&P500
List = CategoryGetSymbols( categoryWatchlist, wlnum ) ;

procedure CreateIndexes( ) //tried replacing procedure with in-line code in main body of code
{
  global List;
  local Symbol,n,IdxName; //tried including Symbol as global
  
  for( n = 0; ( Symbol = StrExtract( List, n ) )  != "";  n++ )
    {
        SetForeign( Symbol ); //tried with and without this
		//if (n == 0) //tried with and without this
		//{
        IdxName = "~bob_all_stocks";
        AddToComposite(Roc( Close, 10),IdxName,"C",27);
        AddToComposite(1,IdxName,"V",27);
        //}
    }
}

CreateIndexes();
t1 = Foreign("~bob_all_stocks","C");
t2 = Foreign("~bob_all_stocks","V");
Filter = 1;
AddColumn(t1,"C",5.5);
AddColumn(t2,"V",5.5);

atcproblem


AddtoComposite or staticvar to build a indicator
#2

@RobertJay Sorry I am not familiar enough with AddToComposite so I hope you will be OK with my attempt at showing something similar using the newer function as of version 6.10 StaticVarAdd.

As well as being multi-threading safe addition “StaticVarAdd generally offers better performance than AddToComposite which was already blazing fast. Single threaded StaticVarAdd may be twice as fast as ATC. With 8 threads running StaticVarAdd may be 4x as fast”

https://www.amibroker.com/guide/afl/staticvaradd.html

Perhaps you could modify this to suit your purposes.

// Using static variable to build a composite ROC reading
// First run a SCAN on the Watch List of interest
// then run the Explore 
// v1.1 changed for SP500 and added column

periods = Param("periods", 10, 5, 100, 1);
ROCval = roc(C,periods);

if (Status("actionEx") == actionScan)
{
	if (Status("stocknum") == 0)
		StaticVarRemove("~ROC*");
	StaticVarAdd("~ROCtotal", ROCval);	
	StaticVarAdd("~ROCcount", 1);
	
}
ROCtotal = StaticVarGet("~ROCtotal");
ROCcount = StaticVarGet("~ROCcount");
AverageROC = (ROCtotal / ROCcount);

// Explore //
//Filter = 1;
Filter = Status ("lastbarinrange" );
AddColumn(ROCcount, "Total Number of Stocks");
AddColumn(ROCval, "ROC");
AddColumn(ROCtotal, "ROCtotal");
AddColumn(AverageROC , "Ave ROC in WL " );

// Chart //
Plot(AverageROC, "AverageROC", colorRed, styleThick);
Plot(ROC(Foreign("$SPX", "C"), 100), "Index ROC(" + periods + ")", colorLightBlue, styleDashed);

Buy= Sell = 0;

image

image

P.S. I post way to often on the forum an hope it doesn’t discourage others from answering your question directly.


#3

Thank you so much portfoliobuilder - that looks fantastic. I’m still very new to AB so I want to study and learn from your code - but I’m hopeful that you provide a great solution.


#4

I noticed an error (after my 10 minutes of edit time) as I changed part of the code to match your “10” day ROC, but in the Plot you will notice that the Index graph has the 100 day ROC, so not a proper comparison with the “AverageROC” plot with its default parameter set at 10.


#5

@RobertJay you should try to understand what does your code do. This code was probably intended to be used as an indicator. For this reason it has SetForeign() function and for this reason it iterates through all symbols from a watchlist. If you use it in exploration you completely don’t need SetForeign() and if you run an exploration on a watchlist containing 500 stocks, this code is executed exactly 499 times more, than it should be. Your code should be executed only once. It can be used in Exploration, but in that case it should be executed only when the very first symbol is processed:

if (Status("stocknum") == 0)
{
  // ......
}

Besides, if you want to create composites using Explorations you need to change AddToComposite() flags. You can find appropriate information here:

http://amibroker.com/guide/afl/addtocomposite.html
http://amibroker.com/guide/a_addtocomposite.html

I really recommend reading this tutorial written by @SpinTrader (Herman) :

http://www.amibroker.org/3rdparty/IntroToAtc.pdf

You can also watch this short video showing how to to create the 52 Week High/Low Index using AddToComposite: https://www.youtube.com/watch?v=Rn2lWFlQs-8

But as @portfoliobuilder said, you can also achieve your goal using Static Variables which in general are way faster than AddToComposite().

Regards.


#6

Milosz - thanks for that great information. In my fairly limited time with AB I’ve concluded that the type of information you provide will be invaluable to me as I learn (and hopefully profit) from my AB journey.


#7

Hi,

I copy pasted the code provided by @portfoliobuilder as a starting point to check the usage of composites created with StaticVarAdd function.

After I ran the code, I noticed that the Total Number of Stocks (stored in ~ROCcount variable) is much higher than the number of symbols in the watchlist.
Using Trace function I noticed that StaticVarAdd is not adding 1 to it’s previous value at each “iteration”. It’s adding the previous value to itself and then adding 1.
So for example in a watchlist of 5 symbols, instead of getting 1+1+1+1+1 (=5) the result I’m getting is the sum of 1+3+7+15+31 (=62).

Does anyone know what might be leading to this result?

thanks very much

Moderator comment: you just NEED to use AmiBroker 6.12 or HIGHER if you want to use scalars with StaticVarAdd


#8

Hi,

For starters I would suggest you post the exact code that you used even though you say you copy and pasted
code provided by @portfoliobuilder. This way we can actually see the entire code.

Please use the code tags as suggested.

Anthony


#9

Hi,

the code I’m using is the same as the one posted by @portfoliobuilder, I just added the trace function to check why I’m getting this results.

// Using static variable to build a composite ROC reading
// First run a SCAN on the Watch List of interest
// then run the Explore 
// v1.1 changed for SP500 and added column

periods = Param("periods", 10, 5, 100, 1);
ROCval = roc(C,periods);

if (Status("actionEx") == actionScan)
{
	if (Status("stocknum") == 0)
		StaticVarRemove("~ROC*");
	StaticVarAdd("~ROCtotal", ROCval);	
	StaticVarAdd("~ROCcount", 1);
	
}
ROCtotal = StaticVarGet("~ROCtotal");
ROCcount = StaticVarGet("~ROCcount");
AverageROC = (ROCtotal / ROCcount);
_TRACE("#AB "+Name()+" "+Roccount);
// Explore //
//Filter = 1;
Filter = Status ("lastbarinrange" );
AddColumn(ROCcount, "Total Number of Stocks");
AddColumn(ROCval, "ROC");
AddColumn(ROCtotal, "ROCtotal");
AddColumn(AverageROC , "Ave ROC in WL " );

// Chart //
Plot(AverageROC, "AverageROC", colorRed, styleThick);
Plot(ROC(Foreign("$SPX", "C"), 100), "Index ROC(" + periods + ")", colorLightBlue, styleDashed);

Buy= Sell = 0;

The results of this expression: _TRACE("#AB "+Name()+" "+Roccount); are the following:
debug_view

The result of exploration in a watchlist with 22 symbols :
exploration

StaticVarAdd instead of adding just 1 at each iteration, is adding it’s the previous value to itself and then adding 1, hence the 1+3+7+15… .
I can’t figure out why this is happening. Is there any setting that could have influence on the calculation?

thanks for your help

kr,
pm

Moderator comment: you just NEED to use AmiBroker 6.12 or HIGHER if you want to use scalars with StaticVarAdd


#10

First of all if someone wants to create any kind of composites he shoud ask himself how is he going to use them. If someone only wants to display it on a chart, all the process of creating a composite can be done directly from a chart. If someone wants to present the results only in exploration he doesn’t need a scan or a chart. Composites can be created at least in 5 different ways:

  1. Directly from a chart (you don’t need neither exploration nor scan in this case) In this case a part of the code which is executed conditionally (every n seconds/minutes/hours or only when it is triggered) iterates through all symbols from a watchlist(s) using Foreign() or SetForeign() and creates a composite.

  2. From an Exploration (you don’t need neither chart nor scan in this case). Composites can be created in two ways:

  • In a way very similar to the above example. The whole process of creating a composite is performed when the very first symbol is processed - if (Status("stocknum") == 0). The code iterates through all symbols from some watchlist(s) using Foreign() or SetForeign() and creates a composite. The disadvantage of this solution is that such code might (but don’t have to) iterate through all symbols twice. First time when the composite is created, second time when the rest of regular exploration (by the definition iterating through all symbols from a watchlist) is performed.

  • Without using Foreign() or SetForeign(). You just run an exploration on a selected watchlist and the process of creating a composite takes place with each symbol that is processed.

  1. From a scan (you don’t need neither exploration nor chart in this case):
  • Analogical to exploration

  • Analogical to exploration

Back in the year 2015 I was calculating (on the Polish stock market) things like New H/L (Ratio), Breadth (Thrust), Absolute Breadth Index, McClellan Oscillator, TRIN Arms Index, Relative performance etc. using AddToComposite() directly from the chart - by iterating through watchlists. I was quite happy with my solution, but then it turned out, that I can achieve exactly the same result using Static Vars instead of AddToComposite(). One of many differences is that AddToComposite() by the definition uses all bars. In case of Static Variables you can use much less bars. If you are a daytrader and want to calculate (every n-seconds) some composite requiring only 200 bars - and you can do it using only 200 bars instead of i.e. 200 000, you will clearly notice substantial speed gains.

One more remark with regards to StaticVarAdd(). It is not something which is absolutely required to create a composite. Creating composites using Static Variables was possible and is still possible without it. StaticVarAdd() was introduced in 6.02 Beta, and its’ main advantage (apart from some others) is that it is multithreading safe addition for static variables that are shared by multiple threads. If only one thread writes the Static variable and many other threads read it, you can, but you don’t have to use it. The most important thing is to understand how does it work. A quote from:
https://www.amibroker.com/guide/afl/staticvaradd.html

In fact when KeepAll is set to False, StaticVarAdd can be seen as the following pseudo code:

EnterCriticalSection
x = Nz( StaticVarGet( "name" ) ); // read exisiting value (and convert Nulls to zero)
x += Nz( value ); // add value to existing
StaticVarSet( "name", x ); // store updated value
LeaveCriticalSection

I began using Static Variables instead of AddToComposite() in most cases, after reading @fxshrat post from the year 2015. It was an eye-opener for me. Fxshrat provided some great examples of how a properly written code creating composites using Static Variables or AddToComposite() should look like. I also understood some advantages of using Static Variables. Here is this thread:

https://groups.yahoo.com/neo/groups/amibroker/conversations/topics/188924

The best advice I can give you all (apart from studying AB documentation and the materials to which I have already provided links above) is to study those Fxshrat’s examples line by line (even if it took you a week) because you will learn a lot. If you ingest this knowledge, creating any composite will be easy. In my opinion custom composites are among those things which might give you an edge on the market :slight_smile:

Regards


Count Num Symbols in Watch List Referencing Name
#11

Thank you @Milosz. Very very helpful.


#12

Thanks @Milosz, I’ll check all the steps you mentioned.
I believe creating the composite using Foreign() within the if (Status("stocknum") == 0) section will solve the issue.

Nevertheless what I’m trying to understand is that I’m not getting the same results as @portfoliobuilder despite using the same code.


#13

Is is highly recommended to use StaticVarAdd instead of other methods.

It is the fastest and least resource consuming

StaticVarAdd advantages:

  • does not use crtiical section anymore - it uses faster read/write locks
  • it only acquires the lock once and looks up the variable map once which is at least 2 times faster than combination of StaticVarGet and StaticVarSet
  • StaticVarAdd is atomic operation - you should really never downplay the importance of thread safety and never downplay the fact that the less time you spend in serial section the more performance you get, see Amhdal’s law

Important: you must use AmiBroker 6.12 or higher if you want to use StaticVarAdd on non-array inputs.


AddToComposite results
#14

@Milosz Great post. @pmxgs I ran the same code against 4 other watch lists and it correctly counted my symbols. I am not an expert, and I don’t know if the code is correct or you are doing something different (run Scan, then run Explore). Perhaps if @Milosz @Vorty @RobertJay or even @Tomasz gave a quick test run of the code they could confirm or refute the problem you are seeing?

@Milosz gave a link to the terrific 15 year old article on AddToComposite that Herman had generously shared with users. It sure seem like it is overdue for that document to be updated with examples using Static Variables. And I completely agree with you that,

In my opinion custom composites are among those things which might give you an edge on the market :slight_smile:

It is beyond my code writing capabilities but if any/all interested users (or @Tomasz ) were willing to contribute then count me in for attempting to produce examples and proof reading / testing code contributed by others. Probably only 5-10 of the regular contributors to this forum are capable and I am grateful for the contributions they have already made. So no offense if they are too busy or feel they have done enough.

The User Guide has one simple example but I am certain the user base would appreciate a few more examples or a Knowledge Base article or two on the use of StaticVarAdd.


#15
  1. Which AB version are you using? Tomasz wrote:

You just NEED to use AmiBroker 6.12 or HIGHER if you want to use scalars with StaticVarAdd

  1. What are your exploration’s settings? If you set the Range to: From-to-dates you might get n-times more results than you expect. Try:
  • Apply to: Filter

  • Range: 1 recent bar(s)

Przechwytywanie


#16

Thanks everyone for the suggestions and explanations.

The issue is related with AB version. I ran the code using 6.10.

KRegards,
pm


#17

I get the correct count of symbols also. I’m using the Apply to = Filter (which points to the S&P watch list).


#18

Tomasz, thank you for another interesting insight information. I wasn’t aware of the fact that StaticVarAdd can be 2 times faster than combination of StaticVarGet and StaticVarSet.

In AB documentation StaticVarAdd was usually compared to AddToComposite:

Thanks to extensive code tuning, StaticVarAdd generally offers better performance than AddToComposite which was already blazing fast. Single threaded StaticVarAdd may be twice as fast as ATC. With 8 threads running StaticVarAdd may be 4x as fast (it does not scale as much as naive person may think, because critical section limits performance due to lock contention).

http://www.amibroker.com/devlog/wp-content/uploads/2015/10/readme6020.html
http://amibroker.com/guide/afl/staticvaradd.html

I didn’t find any direct comparison to StaticVarSet/StaticVarGet.

That is another reason why StaticVarAdd should be used whenever possible. :+1:

One small remark. StaticVarAdd only adds some values, so probably there are some situations in which StaticVarSet/StaticVarGet is more flexible and gives more possibilities - for example when modifying some values already stored in Static Variables…


#19

Thanks all of you recommendation. But the difference comes probably from my database, if I compare this with Excel. I think it is not correct calculate the composite price from the roc, if in the database you have missing prices, for example the last price of only one symbol of Friday is missing.
The single roc is calulate correct. But if some days missing in the pricearray for example in my indices database. Then the Average is calculate with different value. If only one symbol has price gap,
than the calculation is different. Correct?
It is a assumption, but can you check this please and how is this better to control?
SensexTelAviv!
excel3excel1|426x134excel2


#20

@Munichtrader I don’t have data holes (that I know of) so I can’t really test anything that helps. But you might consider adding a condition that prevents bars without data from being added. Perhaps something like,

StaticVarAdd( "~ROCCount", NOT IsNull( RSIBelow ) );

But your initial post was about the RSI and I think you were directed here by @Milosz to see the code and modify it to suit your needs.

Below I have made an attempt to use StaticVarAdd to create a count of number of symbols in a WL that meet a threshold for being below a certain RSI level. Test it out and see if it is accurate.

// Building a composite RSI reading for a WL
// Using StaticVarAdd and instead of SCAN, run EXPLORE
// Compare it to Herman's 2002 document on using AddToComposite

// pick watchlist in AA window
wlnum = GetOption( "FilterIncludeWatchlist" );
List = CategoryGetSymbols( categoryWatchlist, wlnum );
TestLevel = 20; // Param("TestLevel", 20, 5, 35, 1);

if( Status( "stocknum" ) == 0 )
{
    // cleanup variables created in previous runs (if any)
    StaticVarRemove( "~RSI*" );
    StaticVarRemove( "~RSICount*" );
	StaticVarRemove ("~RSIBelow*");
	 
    for( n = 0; ( Symbol = StrExtract( List, n ) )  != "";  n++ )
    {
        SetForeign( symbol );
        RSIVal = RSI( 3 ); // put in the RSI you are interested in
        //TestLevel = 20; // this line seems to work here or above - see line 8
        RSIBelow = RSIVal < TestLevel;
        StaticVarAdd( "~RSIBelow", RSIBelow );
        StaticVarAdd( "~RSICount", NOT IsNull( RSIBelow ) );

        RestorePriceArrays();
    }
}

svRSIBelow = StaticVarGet( "~RSIBelow" );
svRSICount = StaticVarGet( "~RSICount" );
PercentBelow = ( svRSIBelow / svRSICount ) * 100;

///////////////
// Explore
///////////////
Filter = 1;
AddColumn( RSI( 3 ), "RSI3", 1.2, colorBlack, IIf( RSI( 3 ) < TestLevel, colorLime, colorDefault ) );
AddColumn( svRSIBelow , "Stocks with RSI Below ", 1.0 );
AddColumn( svRSICount, "Total Number of Stocks", 1.0 );
AddColumn( PercentBelow, "PercentBelow", 1.1, colorDefault, colorLightYellow );


///////////////
// Chart
///////////////
Plot( PercentBelow, "PercentBelow", colorRed, styleThick );

The output I get looks like this on today’s Dow 30 stocks.

image

If you find something useful for building a trading signal from this, I’d be interested in hearing back from you. Good luck.

P.S. @Tomasz look at the odd result for DD ? If anyone can explain that to me I would love to hear it.