TimeFrameSet and staticvars

[bobptz]

Please consider the following code:

TimeFrameSet(in5minute);

hh = High;
bb = BarIndex();
dd = DateTime();

StaticVarSet("_xxHigh1",Hh);
StaticVarSet("_xxBarc1",BB);

SetChartOptions( 0, chartShowArrows | chartShowDates );
_N( Title = StrFormat( "{{NAME}} - {{INTERVAL}} {{DATE}} Open %g, Hi %g, Lo %g, Close %g (%.1f%%) {{VALUES}}", O, H, L, C, SelectedValue( ROC( C, 1 ) ) ) );
Plot( C, "Close", ParamColor( "Color", colorDefault ), styleNoTitle | ParamStyle( "Style" ) | GetPriceStyle() );

printf("Close = "+C+"\n");
printf("Datetime = "+DateTimeToStr(selectedvalue(DateTime())));

TimeFrameRestore();

StaticVarSet("_xxHigh2",High);
StaticVarSet("_xxBarc2",BarIndex());

My understanding is that the static vars that store compressed arrays (like _xxHigh1) will first be automatically expanded and then stored to shared memory.

Am I right?

Because when I try to retrieve them from another window, they seem to be expanded.

[bobptz]

Here is an example where I have difficulty to pass a compressed array staticvar from script1 to script2. Both scripts are in 1minute period, but script1 uses TimeFrameset(in5minute) to compress the array.

script1:

TF = in5Minute;
TimeFrameSet(TF);
HighComp = High;
StaticVarSet("_HighComp",HighComp);
SetChartOptions( 0, chartShowArrows | chartShowDates );
//_N( Title = StrFormat( "{{NAME}} - {{INTERVAL}} {{DATE}} Open %g, Hi %g, Lo %g, Close %g (%.1f%%) {{VALUES}}", O, H, L, C, SelectedValue( ROC( C, 1 ) ) ) );

_N( Title = StrFormat( "MyTF - Interval: "+Interval()+ "; DateTime: " + DateTimeToStr(selectedvalue(DateTime())) +
" Open %g, Hi %g, Lo %g, Close %g (%.1f%%) {{VALUES}}", O, H, L, C, SelectedValue( ROC( C, 1 ) ) ) );
Plot( C, "Close", ParamColor( "Color", colorDefault ), styleNoTitle | ParamStyle( "Style" ) | GetPriceStyle() );

TimeFrameRestore();
SetBarsRequired( sbrAll, sbrAll ); 

script2:

HighComp = StaticVarGet("_HighComp");

SetChartOptions( 0, chartShowArrows | chartShowDates );
_N( Title = StrFormat( "{{NAME}} - {{INTERVAL}} {{DATE}} Open %g, Hi %g, Lo %g, Close %g (%.1f%%) {{VALUES}}", O, H, L, C, SelectedValue( ROC( C, 1 ) ) ) );
Plot( C, "Close", ParamColor( "Color", colorDefault ), styleNoTitle | ParamStyle( "Style" ) | GetPriceStyle() );

SetBarsRequired( sbrAll, sbrAll );

Both scripts (in debug mode) correctly report the Highs of the last 3 bars: 1.38219, 1.38225 and 1.38338.
The 4th bar from the end (index=11487) is reported to have a high of 1.38354 in script1. However in script2 it is reported to have 1.38338. It seems that the staticvar _HighComp is Expanded in script2. But this does not happen for the last 2 values.

Anybody can help with this issue?

[Tomasz]

User error.

Static vars should ONLY be created in ONE interval, the FINEST (base) interval and retrieved in any other required interval as StaticVarGet essentially compresses static var interval to desired one by matching timestamps. Read http://www.amibroker.com/f?staticvarget for details

Static variables are stored WITH timestamps on every bar. It allows them to be used in any OTHER interval because StaticVarGet knows how to match timestamps.

If you do what you do (i.e placing TimeFrameSet-compressed array without calling required TimeFrameExpand into static variable) the timestamps stored are wrong (read sections http://www.amibroker.com/guide/h_timeframe.html blocks marked in orange - especially the last one - the text in those orange blocks specifically tells you that you are doing things that you should not be doing)

[bobptz]

Hi Tomasz

Static vars should ONLY be created in ONE interval, the FINEST (base) interval and retrieved in any other required interval as StaticVarGet essentially compresses static var interval to desired one by matching timestamps. Read http://www.amibroker.com/f?staticvarget for details

I was not aware of this restriction. I did go through the documentation ( http://www.amibroker.com/f?staticvarget ) and this is not mentioned here. Instead it states:

c) when you read static variable in a different interval that it was originally stored, static variables perform padding/synchronization and time compression/decompression automatically in a similar way as foreign, however Foreign compresses data always from base-time interval, while static variables operate on previously stored interval, hence result may differ. For example, if previously stored data was in daily interval, and you read such static variable in intraday chart, you will see essentially flat lines for each day, representing static data from daily interval. 

So from this part I understand that it is ok to store in higher timeframe and retrieve in lower timeframe. Of course I will see flat lines in the lower TF graph, but right now all I am doing is testing the functionality.

Please tell me if I understood it wrong as I am even more confused now.

BTW, I ignored the "Foreign" part as I never use it and I don;t know how it works. I hope it does not make any difference to our discussion.

The code I presented above is for test purposes only. I just wanted to transfer an array between 2 different charts using different timeframes. I wanted to see how this "static variables perform padding/synchronization and time compression/decompression automatically" works and how I could use it.

My ultimate goal is to create a static var in say 5min interval through an analysis window (operating from 1min period) that does Timeframecompress(in5minute) and then retrieve it from a chart that has a period of 5min. So you see, there is no room for TimeFrameExpand here. My logic Somehow this failed so I went to a much simpler script to locate the problem. I will do a more real life test and present it here today.

[Tomasz]

That is not restriction. That is common sense. It saves from doing unnecessary work and using extra memory for no reason.

If you store static variable in say 1-minute resolution you get retrieve them in ANY N-minute, N-hour, day or whatever longer interval just by calling StaticVarGet. It is as simple as that. StaticVarGet picks bars that MATCH current timeframe time stamps.

5min interval through an analysis window (operating from 1min period)

That is where your problem is. You are making things overly complicated.

If you want 5-minute interval, switch PERIODICITY to 5-minute. That's all. No TimeFrame functions needed.

You did not read the ORANGE BOXES in http://www.amibroker.com/guide/h_timeframe.html carefully enough. Re-read them. Slowly, carefully. Every word in this highlighted areas is important. You can't skip that. You are doing ALL mistakes listed in orange boxes. Really I don't know how much emphasis, large/bold font or blinking text is needed to make people READ the text. Orange boxes already say it LOUD AND CLEAR that a) TimeFrameSet is NOT the same as changing periodicity, b) If you use TimeFrameSet you MUST use TimeFrameExpand before making use of any compressed result, in your case before storing static var - it is all very well explained in the tutorial

[bobptz]

Hello Tomasz

Here is some more insight into the real project:

The analysis window has periodicity 1min (which happens to be the base interval of the database). The script does calculations in FOUR different timeframes, for example: 1min, 5min, 1hour, 1Day. This is done with the commands Timeframeset/Timeframerestore. It creates 4 sets of static variables (for the 4 different timeframes), many of them are arrays. The names are dynamic (containing the interval), therefore I have to use the old style naming of staticvars.

So for example the script is :

// we are in 1min TF
general_process();
save_static_vars();  

// go to 5min
Timeframeset(in5minute);
general_process();
save_static_vars();
Timeframerestore();  

// go to 1Hour
Timeframeset(inHourly);
general_process();
save_static_vars();  
Timeframerestore();

// go to Daily
Timeframeset(inDaily);
general_process();
save_static_vars();  
Timeframerestore();

At the same time I have 4 charts open with the following intervals (through chart periodicity): 1min, 5min, 1hour, 1Day. These 4 charts read their matching staticvars.

I assumed that when the analysis window saves a staticvar array after the command Timeframeset(in5minute), then the 5min chart can easily do a staticvarget and read it without any trouble, without the need for timeframeexpand. Please tell me if I wrong here.

I do know that this last statement seems to go against the tutorial, against the orange box, but I don’t know how to add Timeframeexpand to the logic, without making the script even more complicated.

For more than a year this setup was working without problems. However recently, after testing the same setup with different intervals, I noticed a problem in the staticvar array alignment of the highest TF chart.

The project is more than 2800 lines of AFL code, plus several plugins. For the last 3 days I am trying to troubleshoot this, I am trying to make very small and simple examples to communicate to you my problem. This is why I wrote: “My ultimate goal is to create a static var in say 5min interval through an analysis window (operating from 1min period) that does Timeframecompress(in5minute) and then retrieve it from a chart that has a period of 5min.”

Please verify for me that my setup is supposed to work as it is, and then I will try to figure an example that shows exactly the problem I am facing.

You are doing ALL mistakes listed in orange boxes.

Let me go over them again, one by one and specifically comment.

1. "IMPORTANT: TimeFrame functions are NOT intended to replace Periodicity setting. To switch periodicity/interval you should only use Periodicity setting. TimeFrame functions are ONLY for formulas that MIX many different intervals at once in single formula."
   Yes, I do fall into this category: "formulas that MIX many different intervals at once in single formula".

2. "IMPORTANT: TimeFrameSet() is NOT an equivalent of the periodicity setting in Analysis Settings. The only use for time-frame functions is when you want to have the trading rules based on MULTIPLE time frames at once. See details in "How it works internally" below."
   Yes, same as #1: "when you want to have the trading rules based on MULTIPLE time frames at once".

3. "PLEASE NOTE that you can only compress data from shorter interval to longer interval. So when working with 1-minute data you can compress to 2, 3, 4, 5, 6, ....N-minute data. But when working with 15 minute data you can not get 1-minute data bars. In a similar way if you have only EOD data you can not access intraday time frames."
   Yes, I am compressing from 1min to higher TimeFrames.

4. "IMPORTANT: TimeFrameExpand IS REQUIRED for any formula that uses TimeFrame* functions. If you don't expand time compressed data you will have incorrect timestamps (see description below in "How it works")."
   Yes, I am doing this wrong. Waiting for insight on how to adjust my logic, as stated above.

5. "Time-frame functions do not change the BarCount - they just squeeze the arrays so you have first N-bars filled with NULL values and then - last part of the array contains the actual time-compressed values.
   This is why it is essential to expand the data back to the original frame with TimeFrameExpand."
   Yes, I know this, I have experimented with this one extensively.

[Tomasz]

In shortest possible explanation:

TimeFrameSet( in5Minute );
.... 
StaticVarSet( "name", TimeFrameExpand( compressed_array, in5Minute ) );

[Milosz]

The analysis window has periodicity 1min (which happens to be the base interval of the database). The script does calculations in FOUR different timeframes, for example: 1min, 5min, 1hour, 1Day. This is done with the commands Timeframeset/Timeframerestore. It creates 4 sets of static variables (for the 4 different timeframes), many of them are arrays. The names are dynamic (containing the interval), therefore I have to use the old style naming of staticvars.

At the same time I have 4 charts open with the following intervals (through chart periodicity): 1min, 5min, 1hour, 1Day. These 4 charts read their matching staticvars.

@bobptz Can I ask you, why don't you just calculate Static variables directly in appropriate Indicator windows (which you use) without timeframe functions and without using Exploration for that? You might easily run some parts of the code only when needed - for example only once a minute or when there is a new bar... I think in this case, you don't need exploration at all.

Besides if you use Exploration to calculate some things in a daily timeframe (but the Periodicity is set to 1-minute) it is very ineffective and slows down the performance a lot. 60 min x 24h = 1440 bars. In extreme cases you might need 1440 times more bars to calculate some variables than if you used daily bars for that. Speed difference might be very significant.

[bobptz]

TimeFrameSet( in5Minute );
....
StaticVarSet( "name", TimeFrameExpand( compressed_array, in5Minute ) );

And then in the 5min chart I will read the staticvar and either it will automatically compress (from the Amibroker engine) or I can do timeframecompress("name", in5minute).

Thank you very much! Eager to test this.

[bobptz]

Can I ask you, why don't you just calculate Static variables directly in appropriate Indicator windows (which you use) without timeframe functions and without using Exploration for that?

I need the process to constantly run in the background, as new bars are coming. If I remember well Tomasz had suggested that I use the Analysis Window, because of the autorun feature of every 1sec. Charts that are not directly visible will not run, to save CPU time.

This script controls almost everything, does all the calculations and it will generate the buy/sell signals. The charts are secondary, they have little code, just for the display.

You might easily run some parts of the code only when needed - for example only once a minute or when there is a new bar... I think in this case, you don't need exploration at all.

Of course, the code runs avery 1 sec and first checks if I have a new 1min bar. Then checks if I have a NEW bar in any of the higher Timeframes. For each new (TF) bar that it identifies, it runs the code and generates the static vars.

Besides if you use Exploration to calculate some things in a daily timeframe (but the Periodicity is set to 1-minute) it is very ineffective and slows down the performance a lot. 60 min x 24h = 1440 bars. In extreme cases you might need 1440 times more bars to calculate some variables than if you used daily bars for that. Speed difference might be very significant.

I see what you mean. You are 100% right.

I only need the highest TF to identify major Support/Resistance lines. I could very easily just run some other subsystem and locate those levels. I could also just deactivate the highest TF and run with only 3 or even 2 TimeFrames. My script is scalable and accepts this.

Later on, I may use the static vars to save recalculation of historical bars. I may only calculate things in the most recent 2-3 bars. But this is the future plan.

Thank you for the insight.

[Milosz]

I need the process to constantly run in the background, as new bars are coming. If I remember well Tomasz had suggested that I use the Analysis Window, because of the autorun feature of every 1sec. Charts that are not directly visible will not run, to save CPU time.

Bob - really, a couple months ago we spent many hours discussing this issue, and I even prepared a short code for you, to prove that charts might be invisible (minimized/hidden) and you can still force timed refreshes (as long as they are actual windows not chart sheets).

QuickAFL and Analysis Window

Apply my code to the chart, In the chart’s Parameters select - “Only when visible” - “No” Minimize the chart Minimize the whole AmiBroker Start browsing the Net or watching “House of Cards” I can assure you, your chart is not in focus You will hear if your formula is executed or not … I did it about half an hour ago (my AmiBroker is minimized) and still can hear that: “This part of the code is executed every n seconds” …

Of course, the code runs avery 1 sec and first checks if I have a new 1min bar.

Running an exploration every 1 second just to check if there is a new bar is a very strange solution for me. In my opinion chart windows are much more suitable for that and you would have both - an updated chart (which probably is timely refreshed anyway) and created static variables only when needed. In this case you also wouldn't need any TimeFrame code. I have coded a similar solution and it works very well for me.

Summing up, in my opinion your solution is probably far from being efficient.

[Tomasz]

I wrote already, that you just read the variable using StaticVarGet. Do NOT call TimeFrameCompress.

[bobptz]

Bob - really, a couple months ago we spent many hours discussing this issue, and I even prepared a short code for you, to prove that charts might be invisible and you can still force timed refreshes (as long as they are actual windows not chart sheets).

Yes, I just went quickly over it. You had given a great solution.

My script will run for Bar-replay, live feed and backtesting. All in one code. I have to check if your version can do this.

[bobptz]

@Tomasz

I tried it and it did not work. Probably I am doing something wrong or have somewhere the wrong setting.

Here is one of the problems I noticed:
I have set the chart period to 1min. See the times at the debugger window. They are in 5min intervals, but they are not rounded. ie the times are 16:04, 16:09, 16:14 etc. What am I doing wrong againt? How can I have them be 16:00, 16:05, 16:10 etc ?

BTW, I just checked and I have set correctly the database settings in File -> Database Settings -> Intraday Settings to Start=00:00, End=23:59.

[bobptz]

After examning older notes, forum posts and Amibroker support tickets I can answer why this happens to DateTimes of compressed bars (compressed through the TimeFrameSet function):
"compressed bars take the datetime of the last lower timeframe bar that is included in the higher timeframe bar."

I know I can use this to produce an array of datetimes with correct alignent:

DateComp = TimeFrameCompress( DateTime(), TF, compressOpen );  

However I do not think this will help me with the original problem.

[bobptz]

@Tomasz

I tested it, but it did not work.

chart1, 1min period:

TF = in5Minute;
TimeFrameSet(TF);
HighComp = High;
HighExpanded = TimeFrameExpand( HighComp, TF, ExpandFirst ) ;
StaticVarSet( "_HighExpanded", HighExpanded);
SetChartOptions( 0, chartShowArrows | chartShowDates );
//_N( Title = StrFormat( "{{NAME}} - {{INTERVAL}} {{DATE}} Open %g, Hi %g, Lo %g, Close %g (%.1f%%) {{VALUES}}", O, H, L, C, SelectedValue( ROC( C, 1 ) ) ) );

_N( Title = StrFormat( "MyTF - Interval: "+Interval()+ "; DateTime: " + DateTimeToStr(selectedvalue(DateTime())) +
" Open %g, Hi %g, Lo %g, Close %g (%.1f%%) {{VALUES}}", O, H, L, C, SelectedValue( ROC( C, 1 ) ) ) );
Plot( C, "Close", ParamColor( "Color", colorDefault ), styleNoTitle | ParamStyle( "Style" ) | GetPriceStyle() );

TimeFrameRestore();
SetBarsRequired( sbrAll, sbrAll );

script2, 5min period:

HighExpanded = StaticVarGet("_HighExpanded");

SetChartOptions( 0, chartShowArrows | chartShowDates );
_N( Title = StrFormat( "{{NAME}} - {{INTERVAL}} {{DATE}} Open %g, Hi %g, Lo %g, Close %g (%.1f%%) {{VALUES}}", O, H, L, C, SelectedValue( ROC( C, 1 ) ) ) );
Plot( C, "Close", ParamColor( "Color", colorDefault ), styleNoTitle | ParamStyle( "Style" ) | GetPriceStyle() );

SetBarsRequired( sbrAll, sbrAll ); 

The column "High" has the correct values of the HIGH from the 5min bars on the chart. The DateTime column also has the correct timestamps of the bars on the chart (the times are nice and rounded to the 5min).

1. In script2, I expected the column HighExpanded to be IDENTICAL with the column High. But they are not.
2. In script1 the last value of HighExpanded is 1.38219. In script2 the last value of HighExpanded is 1.38225. At least I would expect the last value of the static var to be the same in both scripts.

Could you please tell me what I am doing wrong?

[bobptz]

Please note that the purpose of all this was to be able to send a staticvar array from a script that does TimeFrameCompress(in5minute) to a chart that has period 5min. And the array to completely align with the bars of this second chart:

1. In script2, I expected the column HighExpanded to be IDENTICAL with the column High. But they are not.

Can this be done?

[awilson]

TF = in5Minute;
TimeFrameSet(TF);
HighComp = High;
TimeFrameRestore();

HighExpanded = TimeFrameExpand( HighComp, TF, ExpandFirst ) ;
StaticVarSet( "_HighExpanded", HighExpanded);
SetChartOptions( 0, chartShowArrows | chartShowDates );
//_N( Title = StrFormat( "{{NAME}} - {{INTERVAL}} {{DATE}} Open %g, Hi %g, Lo %g, Close %g (%.1f%%) {{VALUES}}", O, H, L, C, SelectedValue( ROC( C, 1 ) ) ) );

_N( Title = StrFormat( "MyTF - Interval: "+Interval()+ "; DateTime: " + DateTimeToStr(selectedvalue(DateTime())) +
" Open %g, Hi %g, Lo %g, Close %g (%.1f%%) {{VALUES}}", O, H, L, C, SelectedValue( ROC( C, 1 ) ) ) );
Plot( C, "Close", ParamColor( "Color", colorDefault ), styleNoTitle | ParamStyle( "Style" ) | GetPriceStyle() );

SetBarsRequired( sbrAll, sbrAll );

Not tested, but TimeFrameExpand MUST be done after TimeFrameRestore

[Milosz]

Bob, I really don't understand your approach. You seem to choose the most complex solution possible.

At the same time I have 4 charts open with the following intervals (through chart periodicity): 1min, 5min, 1hour, 1Day. These 4 charts read their matching staticvars.

If you already use 4 charts - each one in a different interval (1 min, 5 min, 1 hour, 1 day), then instead of using exploration or using TimeFrameSet/TimeFrameRestore/TimeFrameExpand code, let each of these charts do the necessary job in its interval.

In such case, instead of this code:

TF = in5Minute;
TimeFrameSet(TF);
HighComp = High;
TimeFrameRestore();

HighExpanded = TimeFrameExpand( HighComp, TF, ExpandFirst ) ;
StaticVarSet( "_HighExpanded", HighExpanded);

... you would just need this:

HighComp = High;

... but if you only store High value in Static vars (your example), then you don't need any Static Vars at all !!! Such code should be simple, light and effective!

Besides such solution would be much more efficient, because if you for example need to calculate MA(H, 200) from daily bars, you would just need 200 daily bars instead of (60x24x200 = 288 000 1 minute bars) - it is especially important - because as I can see, you are forcing the code to use all bars!!!

Additionally if you spread some really demanding calculations between 4 different chart windows (there is really no such need if the code is properly written - I've never needed such gimmicks), you might in some cases use 4 CPU threads/cores instead of 1.

[bobptz]

Not tested, but TimeFrameExpand MUST be done after TimeFrameRestore

Yes, you are right, Thank you so much.

Only the last value of HighExpanded in script2 does not match. It should be 1.38219. Instead it is 1.38225. Why is that?

The last bar is critical.