Importing proprietary indicators from Tradestation into Amibroker

I have some proprietary indicators in Tradestation that I need to get into Amibroker in 60min, Daily and Weekly intervals. I built a data-export app in TS and now have the SP500 constituents' data for these indicators, bar-by-bar, for all three time frames, going back for 10+ years. I then built an exploration script in Amibroker that reads all of these files and assigns each indicator value bar-by-bar to staticvars. The key being something like: "myIndicator_AAPL_60min_DateNum_TimeNum" and the value being the indicator value for that particular bar. I then built an AB reader/backtester script that can assign these staticvars to arrays for bar-by-bar analysis and backtesting.

The issue is that it works for a handful of symbols ok, but populating the staticvars from files for the entire SP500 takes hours and also AB becomes really bulky and cumbersome. I've check my system resources and AB grows to be many gigs. I am thinking this solution will not work now....

I have read about other people trying to do this but have found no solutions other than Aux1 and Aux2, which are not enough fields. I need about twenty fields, not two.

Any suggestions?

You are abusing static variables. Static variables are not "endless" storage. They are kept in RAM and you must have enough RAM to hold them. If you don't then paging file starts to be used and this costs lots of CPU cycles.

Go to Tools->Performance Monitor and make sure that memory used by static variables is NOT higher than 25% of RAM. (Other RAM is needed for operating system, data cache and other programs running on the system)

image

Static variables ARE NOT provided for keeping imported data. You should import to and keep the data in the database, not in static variables. You can have any number of "fields". You should just use artificial tickers and use ALL fields, i.e OHLCV, OI, Aux1, Aux2.

1 Like

What about the mixed intervals? Can I import 60min, Daily, Weekly files separately into the Database? The indicators' values are different for all three so they have to be separated.

You can create any number of artificial tickers in any interval such as MYDATA_Daily, MYDATA_60Minute, MYDATA_WEEKLY.

Yes, I got it to work. And I see that there are options while using Foreign to fill in holes, which is working nicely for the larger time-frames.

I had it working fine using 15min data, but when I move to hourly data things blow up.

First a little background on my data. I am importing it from Tradestation, so I am using end-of-bar time stamps. I live in the Central North American time zone so the US stock market closes at 15:00:00 my time. But I had to export my data with the last bar of the day being 15:30:00 because when it was set at 15:00:00 Amibroker was compressing the last two bars together. So for hourly data it had six bars in the day instead of the correct seven bars in the day until I did this last-bar-of-day time adjustment.

Now for the proprietary indicators I have them in the form of hourly, daily and weekly data as you have described above, with the symbol being something like "AAPL_Hourly_MjrSup". I have set each of these datasets to end at 15:30:00 also.

I have reduced my thousands of lines of code into a very small snipit and reproduced the problem to share here. The issue now is that the following code gets stuck in an endless loop when the for loop uses BarCount. But when I change BarCount to a number like 1500 it does not. What do you think is going on here?


_SECTION_BEGIN("Price");
SetChartOptions(0,chartShowArrows|chartShowDates);
_N(Title = StrFormat("{{NAME}} - {{INTERVAL}} {{DATE}} Open %g, Hi %g, Lo %g, Close %g (%.1f%%) {{VALUES}}", O, H, L, C, SelectedValue( ROC( C, 1 ) ) ));
Plot( C, "Close", ParamColor("Color", colorDefault ), styleNoTitle | ParamStyle("Style") | GetPriceStyle() ); 
_SECTION_END();


doTrace = 1;
bi = BarIndex();
dn = DateNum();
tn = TimeNum();

printf( "DT: " + NumToStr( dn, 1.0, False, False ) + "_" + NumToStr( tn, False, False) + "\n" ); 

UseMjrSup1 = 1;

Daily_MjrSup1 = IIf( UseMjrSup1, Foreign( Name() + "_Daily_MjrSup", "O" , 2), 0);

for ( i = 1; i < BarCount; i++ )
{
	dn_tn = NumToStr(dn[i], 1.0, False, False) + "_" + NumToStr(tn[i], 1.0, False, False);
	dn_tn_i = "DT: " + dn_tn + " i: " + NumToStr(i,1.0, True, False) + " ||||| " ;
		
	if ( doTrace AND i = 21094 ) { _TRACE(dn_tn_i + " Daily_MjrSup1[i-1]: " + NumToStr(Daily_MjrSup1, 1.2)); }
}

This is wrong

if ( doTrace AND i = 21094 )

correct equality check:

if ( doTrace AND i == 21094 )
1 Like

One additional note concerning using Static variables in general (I'm not referring to this particular situation, as Tomasz has shown you the proper solution). Starting from AB 6.10 Static vars can be compressed, which (in some cases) might allow you to save a lot of RAM. Check if you use compression when storing lots of data in Static vars.

Starting from version 6.10 there is a new parameter compressionMode that decides whenever given variable will be compressed or not.By default only persistent static variables will be compressed (cmDefault). You can turn it off completely compressionMode = cmNever, or turn it on for persitent and non-persistent variables using compressionMode = cmAlways

Compression is done by removing repeated values from the sequence as repeated values are restored when doing StaticVarGet. Compression is NOT compatible with non-aligned mode of StaticVarGet. If compressed array is retrieved by StaticVarGet with align=False, then repeated values found in original array would not be retrieved. Turning compression on slows down StaticVarSet (as it needs to do some extra processing), but does not affect performance of other functions, so StaticVarGet is equally fast with or without compression.

https://www.amibroker.com/guide/afl/staticvarset.html