Bad formula, was: Amibroker CPU usage

Hi,

i have noticed that amibroker is starting to use more and more CPU usage (i have 16g ram and it's a del xps 13 i77560u 2.4gz ), it's up around 50% now, is there a way to check what scripts or plugins are using the resoures?
i am running iqfeed data tick.

Cheers
Jon

I'm sure, that you are using some very inefficient AFLs, and they are to blame. If you want to get some insights how efficient your code is, (among other ways) you can make use of the below functionalities:

  1. Click Tools --> Preferences --> Miscellaneous --> Display chart timing (advanced). As a result you will see some statistics at the bottom of each chart:

Image%201

  1. Go to AFL Formula Editor and click Tools --> Code Check & Profile. You will see a detailed summary similar to this one:

Image%202

  1. Use Tools --> Performance Monitor:

1

I also recommend reading these articles:

https://www.amibroker.com/guide/x_performance.html
https://www.amibroker.com/guide/h_rtsource.html
https://www.amibroker.com/guide/h_multithreading.html

3 Likes

Ami

ami2

My time per calls are much much greater then the example. do you think thats the problem?

ami3

ok found this one huge call time

_SECTION_BEGIN( "MPLite" );
// - there was a plot limit call introduced after 5.41.0 beta
//   i.e. "•Warning 502: Calling Plot()/PlotOHLC over 500 times is displayed in indicator in runtime to prevent abuse"
// - uses GFX for major calls of plot only if AB version newer than 5.40 - this only delays reaching the plot limit
// http://www.***********.com/amibroker/2961-market-profile-amibroker-new-13.html

Version (5.80);
PS_SOLID=0;
PS_DASH=1;
PS_DOT=2;
PS_DASHDOT=3;
PS_DASHDOTDOT=4;
PS_NULL=5;
PS_INSIDEFRAME=6;

function Lastthursday()
{
 Daysinmonth=
    IIf(Month()==2, IIf(Year()%4==0 AND Year()%100!=0,29,28), 
    IIf(Month()==4 OR Month()==6 OR Month()==9 OR Month()==11,30,31));
 
 return IIf(Daysinmonth-Day()<7 AND DayOfWeek()==4,1,
        IIf(Daysinmonth-Day()<8 AND DayOfWeek()==3 AND Ref(DayOfWeek(),1)!=4 AND Day()!=Daysinmonth,1,0));
 
}


procedure GfxPlotLine( ix0, iy0, ix1, iy1, iColor, iWidth, iStyle)
{
    GfxSetBkMode(1);
    GfxSelectPen( iColor, iWidth, iStyle );
    GfxMoveTo( ix0, iy0 );
    GfxLineTo( ix1, iy1 );
}


PlotOHLC( O, H, L, C, "Price", IIf( C > O, colorGreen, colorRed ), styleCandle );

FirstVisibleBar = Status( "FirstVisibleBar" );
Lastvisiblebar = Status( "LastVisibleBar" );

totalVisible = Lastvisiblebar - FirstVisibleBar;


//Den = Param("Density", 1, 0.1, 10, 0.1);
percent = Param( "Value Area", 70, 1, 100, 1 );
Type = ParamList( "Type", "Price Profile|Volume Profile" );
Period = ParamList( "Base", "Hourly|Daily|Weekly|Monthly|Lastthursday|Yearly", 1 );
x_scale = Param( "Horizontal_scale", 2, 0, 10, 0.1 );
EnMP2 = ParamStyle( "Style", styleLine | styleNoLabel, maskAll );

EnIB = ParamToggle( "Show Initial Balance",  "No|Yes", 0 );
IBBars = Param( "Initial Balance Bars", 2, 1, 10, 1 );
ViewYvalues = ParamToggle( "Show Yesterdays Values", "No|Yes", 1 );
ViewVlines = ParamToggle( "Show Vertical Base Lines", "No|Yes", 1 );
Viewvalues = ParamToggle( "Show Values", "No|Yes", 0 );
ViewVpoc = ParamToggle( "Show Virgin POC", "No|Yes", 1 );
ViewTPO = ParamToggle( "Show TPO Count", "No|Yes", 0 );

Color_Above_VA = ParamColor( "Color_Above_VA",  colorGrey40 );
Color_VA = ParamColor( "Color_VA", colorBlueGrey );
Color_Below_VA = ParamColor( "Color_Below_VA", colorGrey40 );
Color_POC_line = ParamColor( "Color_POC_Line", colorYellow );

color_YVAH = ParamColor( "YVAH", colorWhite );
color_YVAL = ParamColor( "YVAL", colorWhite );
color_YPOC = ParamColor( "YPOC", colorYellow );

IBColor = ParamColor( "IB Color", colorWhite );
IBstyle = ParamStyle( "IB style", styleLine, maskAll );
Color_Virgin_POC = ParamColor( "Virgin Poc Color", colorYellow );
Color_Base_Line = ParamColor( "Base Line Color", colorDarkGrey );

switch (Period)
  {

    case "Yearly":    
        BarsInDay = BarsSince(Year() != Ref(Year(), -1));
        tf = inMonthly;
        break;

    case "Monthly":    
        BarsInDay = BarsSince(Month() != Ref(Month(), -1));
        tf = inMonthly;
        break;
    
    
    case "Weekly":
        BarsInDay = BarsSince(DayOfWeek() < Ref( DayOfWeek(), -1 ));
        tf = inWeekly;
        break;
    
    case "Hourly":
        BarsInDay = BarsSince(Hour() != Ref(Hour(), -1));
        tf = inHourly;
        break;
    
    case "Daily":
        BarsInDay = BarsSince(Day() != Ref(Day(), -1));
        tf = inDaily;
        break;
    
  }
  

  if(Period=="Lastthursday"  )
  {
    BarsInDay = BarsSince(Lastthursday()==0 AND Ref(Lastthursday(), -1)==1);
    bot1=ValueWhen(BarsInDay==0,L);
    Bot2 = ValueWhen(Ref(BarsInDay)==1 OR BarIndex()>BarCount-2 ,LLV(L,BarsInDay),0);
    bot=Min(bot1,bot2);
    top1=ValueWhen(BarsInDay==0,H);
    Top2 =ValueWhen(Ref(BarsInDay,1)==1 OR BarIndex()>BarCount-2,HHV(H,BarsInDay),0);
    top=Max(top1,top2);
    Vol = ValueWhen(Ref(BarsInDay,1)==1 OR BarIndex()>BarCount-2 ,Sum(V,BarsInDay),0);
  }
  else
  {
    Bot = TimeFrameGetPrice("L", tf, 0);
    Top = TimeFrameGetPrice("H", tf, 0);
    Vol = TimeFrameGetPrice("V", tf, 0);  
  } 

CurTop = HHV( H, BarsInDay + 1 );
Curbot = LLV( L, BarsInDay + 1 );

Range =  Top - Bot;
coverage = LastValue( Ref( ATR( 10 ), -1 ) );

den = ( coverage / 10 );
//den=1;

multiplier = IIf( round( totalVisible ) / 300 < 1, 1, round( totalVisible ) / 300 );

if ( ViewVlines)
    Plot( BarsInDay == 0, "", Color_Base_Line, styleHistogram | styleOwnScale | styleLine | styleNoLabel );


relTodayRange = 0;

x = 0;
basey = 0;
basex = 0;
newday = 0;
total = 0;
shiftup = 0;
shiftdn = 0;
//Line = Null;
Voloumeunit = 0;


 GfxSetOverlayMode( 1 ); 
 GfxSetCoordsMode( 1 ); // bar/price mode (instead of pixel) 

for (  i = IIf( FirstVisibleBar - 100 > 1, FirstVisibleBar - 100, FirstVisibleBar ); i < Lastvisiblebar AND i < BarCount - 1 ; i++  )
{
    if ( BarsInDay[i] == 0 )
    {
        t = BarsInDay[i-1];


        if ( EnIB )
        {
            IBH = HHV( H, IBBars );
            IBL = LLV( L, IBBars );
            /*
            Line1 = LineArray( basex, IBH[basex+IBBars-1], i, IBH[basex+IBBars-1] );
            Plot( Line1, "", IBColor, IBstyle );
            Line2 = LineArray( basex, IBL[basex+IBBars-1], i, IBL[basex+IBBars-1] );
            Plot( Line2, "", IBcolor, IBstyle );
            */
            GfxPlotLine(basex, hY=IBH[basex+IBBars-1], i, hY,
                        IBColor, 1, PS_SOLID);    
            GfxPlotLine(basex, hY=IBL[basex+IBBars-1], i, hY,
                        IBColor, 1, PS_SOLID);                
            
        }

        //////////////////////////////////
        poc = 0;
        pocj = 0;
        midrange = int( relTodayRange / 2 ) + 1;

        for ( j = 1; j <= relTodayRange + 1 ; j++ )
        {
            if ( poc < x[j] )
            {
                poc = x[j];
                pocj = j;
            }
            else
                if ( poc == x[j] )
                {
                    if ( abs( midrange - j ) < abs( midrange - pocj ) )
                    {
                        poc = x[j];
                        pocj = j;
                    }
                }

        }

        for ( n = 1; n <= relTodayRange; n++ )
            total[n] = x[n] + total[n-1];

        Value_area = ( total[relTodayRange] * percent ) / 100;

        for ( a = 1; a <= relTodayRange; a++ )
        {
            if ( pocj - a > 0 AND pocj + a < relTodayRange )
            {
                if ( poc + total[pocj+a] - total[pocj] + ( total[pocj] - poc ) - total[pocj-( a+1 )] >= Value_area )
                {
                    shiftup = shiftdn = a;
                    break;
                }
            }
            else
                if ( pocj - a < 1 )
                {
                    if ( poc + total[pocj+a] - total[pocj] + ( total[pocj] - poc ) >= Value_area )
                    {
                        shiftup = a;
                        shiftdn = pocj;
                        break;
                    }
                }
                else
                    if ( pocj + a > relTodayRange )
                    {
                        if ( poc + total[relTodayRange] - total[pocj] + ( total[pocj] - poc ) - total[pocj-( a+1 )] >= Value_area )
                        {
                            shiftup = floor( relTodayRange ) - pocj;
                            shiftdn = a + 1;
                            break;
                        }
                    }
        }


        //Vah = LineArray( baseX[i], baseY + ( pocj + shiftup ) * den, i, baseY + ( pocj + shiftup ) * den );
        //Val = LineArray( baseX[i], baseY + ( pocj - shiftdn ) * den, i, baseY + ( pocj - shiftdn ) * den );
        //pocline = LineArray( basex, basey + pocj * den, basex[i] + poc, basey + pocj * den );
        
        
        
        if ( ViewVpoc)
        {
            Virginpoc = basey + pocj * den;
            newi = Null;

            for ( j = i; j <= ( BarCount - 1 ) ; j++ )
                if ( L[j] < Virginpoc AND H[j] > Virginpoc )
                {
                    newi = j;
                    break;
                }
                else
                    newi = BarCount;

            Plot( LineArray( basex[i], hY=basey + pocj*den, newi, hY), "", Color_Virgin_POC, styleNoRescale );
            
            
        }

        

        if ( ViewYvalues)
        {
           //Vahn = LineArray( i, baseY + ( pocj + shiftup ) * den, i + t, baseY + ( pocj + shiftup ) * den );
           //pocn = LineArray( i, baseY + ( pocj ) * den, i + t, baseY + ( pocj ) * den );
           //Valn = LineArray( i, baseY + ( pocj - shiftdn ) * den, i + t, baseY + ( pocj - shiftdn ) * den );

            GfxPlotLine( i, baseY + ( pocj + shiftup ) * den, i + t, baseY + ( pocj + shiftup ) * den, 
                 color_YVAH, 0, PS_DOT);
                 
            GfxPlotLine( i, baseY + ( pocj ) * den, i + t, baseY + ( pocj ) * den, 
                 color_YPOC, 0, PS_DOT);     
                 
            GfxPlotLine(i, baseY + ( pocj - shiftdn ) * den, i + t, baseY + ( pocj - shiftdn ) * den, 
                 color_YVAL, 0, PS_DOT);
                     
            //Plot( Vahn, "", color_YVAH, styleDashed | styleNoRescale );
            //Plot( Valn, "", color_YVAL, styleDashed | styleNoRescale );
            //Plot( pocn, "", color_YPOC, styleDashed | styleNoRescale );
            
            
        }


        if ( ViewTPO )
        {
            nnn = HHV( H, BarsInDay );
            PlotText( "" + ( total[relTodayRange] - total[pocj] ), basex[i], nnn[i], colorLightGrey );
            PlotText( "" + ( total[pocj-1] ), basex[i], basey - den, colorLightGrey );
        }

        if ( Viewvalues)
        {
            PlotText( "" + ( ( basey + pocj*den ) ), baseX[i], basey + pocj*den, colorWhite, colorDarkGrey );
            PlotText( "" + ( ( baseY + ( pocj + shiftup )*den ) ), baseX[i], baseY + ( pocj + shiftup )*den, colorWhite, colorDarkGrey );
            PlotText( "" + ( ( baseY + ( pocj - shiftdn )*den ) ), baseX[i], baseY + ( pocj - shiftdn )*den, colorWhite, colorDarkGrey );
        }

        //-- Plot Market Profile
        for ( p = 0; p <= relTodayRange + 1; p = p + multiplier )
        {
            GfxPlotLine( baseX, hY=baseY + p * Den, baseX + x[p], hY, 
                            IIf( p >  ( pocj + shiftup ), Color_Above_VA, 
                            IIf( p <= ( pocj + shiftup ) AND p >= ( pocj - shiftdn ), Color_VA, 
                              Color_Below_VA )), 1, PS_SOLID);
                              
        }
        
        //-- Plot PoC
        GfxPlotLine( baseX, hY = basey + pocj * den, basex[i] + poc, hY, 
                     Color_POC_Line, 1, PS_SOLID);                  
        
                          
        
                     
///////////////////
        basex = 0;
        x = 0;
        Basex = i;
        baseY = Bot[i];
        relTodayRange = Range[i] / Den;
        Voloumeunit = Vol[i] / LastValue( BarsInDay );

    }

    for ( j = 0; j <= relTodayRange ; j++ )
    {
        if ( L[i] <= Bot[i] + j*Den AND H[i] >= Bot[i] + j*Den  )
        {

            if ( Type == "Price Profile" )
            {
                x[j] += x_scale;
            }
            else
                if ( Type == "Volume Profile" )
                {
                    x[j] += round( V[i] / Voloumeunit ) + 1;
                }

        }
    }
}


if ( EnIB )
{
    IBH = HHV( H, IBBars );
    IBL = LLV( L, IBBars );
    Line1 = LineArray( basex, IBH[basex+IBBars-1], i, IBH[basex+IBBars-1] );
    Plot( Line1, "", IBColor, IBstyle );
    Line2 = LineArray( basex, IBL[basex+IBBars-1], i, IBL[basex+IBBars-1] );
    Plot( Line2, "", IBColor, IBstyle );
}

//////////////////////////////////
poc = 0;
pocj = 0;
midrange = int( relTodayRange / 2 ) + 1;

for ( j = 1; j <= relTodayRange + 1 ; j++ )
{
    if ( poc < x[j] )
    {
        poc = x[j];
        pocj = j;
    }
    else
        if ( poc == x[j] )
        {
            if ( abs( midrange - j ) < abs( midrange - pocj ) )
            {
                poc = x[j];
                pocj = j;
            }
        }

}

for ( n = 1; n <= relTodayRange; n++ )
    total[n] = x[n] + total[n-1];

Value_area = ( total[relTodayRange] * percent ) / 100;


for ( a = 1; a <= relTodayRange; a++ )
{
    if ( pocj - a > 0 AND pocj + a < relTodayRange )
    {
        if ( poc + total[pocj+a] - total[pocj] + ( total[pocj] - poc ) - total[pocj-( a+1 )] >= Value_area )
        {
            shiftup = shiftdn = a;
            break;
        }
    }
    else
        if ( pocj - a < 1 )
        {
            if ( poc + total[pocj+a] - total[pocj] + ( total[pocj] - poc ) >= Value_area )
            {
                shiftup = a;
                shiftdn = pocj;
                break;
            }
        }
        else
            if ( pocj + a > relTodayRange )
            {
                if ( poc + total[relTodayRange] - total[pocj] + ( total[pocj] - poc ) - total[pocj-( a+1 )] >= Value_area )
                {
                    shiftup = floor( relTodayRange ) - pocj;
                    shiftdn = a + 1;
                    break;
                }
            }
}

Vah = LineArray( baseX[i], baseY + ( pocj + shiftup ) * den, i, baseY + ( pocj + shiftup ) * den );
Val = LineArray( baseX[i], baseY + ( pocj - shiftdn ) * den, i, baseY + ( pocj - shiftdn ) * den );
pocline = LineArray( basex, basey + pocj * den, basex[i] + poc, basey + pocj * den );

if ( ViewTPO)
{
    PlotText( "" + ( total[relTodayRange] - total[pocj] ), basex[i], top[i], colorLightGrey );
    PlotText( "" + ( total[pocj-1] + x_scale ), basex[i], basey - den, colorLightGrey );
}

if ( Viewvalues )
{
    PlotText( "" + ( ( basey + pocj*den ) ), baseX[i], basey + pocj*den, colorWhite, colorDarkGrey );
    PlotText( "" + ( ( baseY + ( pocj + shiftup )*den ) ), baseX[i], baseY + ( pocj + shiftup )*den, colorWhite, colorDarkGrey );
    PlotText( "" + ( ( baseY + ( pocj - shiftdn )*den ) ), baseX[i], baseY + ( pocj - shiftdn )*den, colorWhite, colorDarkGrey );
}



for ( p = 0; p <= relTodayRange + 1; p = p + multiplier )
{
        GfxPlotLine( baseX,        hY=baseY + p * Den, 
                     baseX + x[p], hY, 
                     IIf( p > ( pocj + shiftup ), Color_Above_VA, 
                     IIf( p <= ( pocj + shiftup ) AND p >= ( pocj - shiftdn ), Color_VA, Color_Below_VA )), 1, PS_SOLID);
}



Plot( pocline, "", Color_POC_Line, styleNoRescale | styleNoLabel );

_SECTION_END(); 

Bingo:

1

It takes your computer over 3 seconds just to calculate these two functions! So as I expected your code is extremely inefficient...

2 Likes

OMG :scream: I've just noticed, that those 3 seconds for HHV + LLV is nothing comparing to your total formula execution time which is (hang on tight): over 2 minutes !!! That's really impressive ... :crazy_face:

1


BTW, no "Thank you", no "Like", no "Solution", apparently you didn't find my help useful ...

3 Likes

There is something wrong with your database settings. Perf monitor shows that youve got 16 million bars per symbol 640MB that would obviously kill any CPU. Lower the number of bars.

This plus double nested loops in your code (meaning billions of steps of inner loop body executed every tick) are reason of your problem.

1 Like

Thanks guys appologies i'm still trying to work it out,
i have roughly 1 million tick bars per day on the es, does this mean i cant hold a month of tick data?

You can start by changing this toy :smiley: :smiley:
Possibilities are many, but limitations exist in this finite context.

@Milosz tested some code on supercomputers, you may want to check that out too :stuck_out_tongue:
just joking.

On a serous note, the performance you want to achieve is directly proportional to the hardware it runs on.

Somebody was running it on Amazon AWS, I don't know how the virtualized instance has fared but one day
you may get conclusive performance reports of what can be done on something like:
m5d.24xlarge https://aws.amazon.com/ec2/instance-types/
VCPU: 96 , RAM: 384 GB and Nvme SSDs.

Cheers, but there are people running a continuous ES contract with more charts + scripts + price ladders and a flood charts on a surface pro 3 flawlessly with ninja trader. i am trying to do half of that .

i have lowered the bars to 10 million (2 weeks) and ill try and work on the above script.

Cheers all

Your own screenshot - Performance monitor - shows 16 million, not 1 million

image

You need to go to File->Database Settings AND DECREASE "number of bars" to 1 million.

And most importantly YOUR CODE IS BAD.

It iterates billions of times inside double nested loops for each tick. REWRITE your own code. Hire somebody who knows how to program in AFL. Your code as it is now is complete trash. You must replace double nested loops with ARRAY processing.
Properly written code for TICK charts should execute below 0.1 second. For that you must use ARRAY processing and must utilize benefits of QuickAFL.

1 Like

Yes it's 1 million quotes (tics) required per day, i have around 15 days of tick data that's about 1/4 timeline of a current contract..

Yes the code is shit we have now worked that out.