Efficiency in the automation of tasks

Vegeta · February 5, 2022, 9:07pm

Hi again!

If I have N systems and I want to be able to run a backtest, an explorer that gets its equity curve and a second explorer that gets its profit table, I must create 3 .APX for each system and then a batch that call the three .APX

Is there any way to avoid doing 3 APX per system?

The backtest formula is logically different for each system, but both the equity curve explorer and the explorer for the profit table are the same for the N systems, the only thing that changes is the selection of the equity curve ticker.

It is becoming unmanageable for me to have 3 APX for each system, imagine having 30 systems would be 90 APX... crazy.

I really appreciate your advice to find a proper solution to manage it

Good night!

fxshrat · February 5, 2022, 9:42pm

You do not need three APX files per system.
Simply use explorer code as include file since it is the same for all systems.

the only thing that changes is the selection of the equity

I don't know what you do there exactly with it but
e.g. make selection dependent on file name of backtest code.

Vegeta · February 6, 2022, 8:14pm

I mean, I have the backtest results:

backtest

This backtest generate a custom equity curve symbol, and then with that equity curve symbol I execute 2 explorers, one for table profit
explorer1

And other explorer to export some results (same equity curve as before)
explorer2

I would like do that for each system... this is my scenario, so...

My little knowledge made me make 3 APX one for each execution... but I'm also duplicating the two explorers for each system because I don't know how to reuse it to export each one separately.

In the end what I need is the export of the backtest, the export of the profit table and the export of the other explorer. But this repeated for each of the systems, and I need 3 different files (exports) by each system.

Do you know a better way?

Thank you very much

fxshrat · February 6, 2022, 8:46pm

Man, do you listen?
I t
I wrote to you already what to do to avoid three apx files.
Use include file (include file contains the two exploration procedures).
Then you need just one APX file not three.
Exploration is the same for all systems.

Create two same equities with different name (since you do not run two explorations at same time).

Then you use batch to load APX, do backtest, set equity symbol name per exploration before each exploration and then run exploration after setting name.

Batch:

Load apx
Backtest
Set equity name
run exploration 1
export to file
set equity name 2 (same equity as before but different name)
run exploration 2
export to file

Single AFL of single APX:

#include <MyExplorations.afl>

/*
    Backtest code with custom equity symbol(s) creation
*/

if ( Name() == "~~~Equity1" )
    RunExploration1();

if ( Name() == "~~~Equity2" )
    RunExploration2();

BTW, backtest report has profit/loss table already!

Vegeta · February 7, 2022, 9:09am

Thank you very much, I tried to do the includes and it works "manually", I mean when I change the equity1 for the first ticker, it explores good and when I select "equity2" it explores another one good. That's fine. But when I try to do it as a batch I have the following problem:

The backtest goes over a watchlist, but from the batch before run the first exploration I put "set current symbol" to the symbol "~~~Equity1" it keeps using the same watchlist with which I saved the APX, so the explorer doesn't run over the ticker that I'm setting in batch.

Can't it be done from the batch to use the current symbol instead of the backtest watchlist?

batch

Thank you very much

fxshrat · February 7, 2022, 10:07am

Here we go again... change of information.

Your picture of upper post was showing single security backtest! See below -> Apply to: Current setting!
So the batch process described was for your single security backtest too.

So if you do backtest over watchlist now... obviously the equity symbols have to be part of that watchlist.
Either added manually once before starting batch or added programmatically via CategoryAddSymbol() after backtest.
Also in that case of portfolio backtest "Set Current Symbol" batch action is not required.

Batch (portfolio backtest):

Load apx
Backtest
Run exploration 1
Export to file
Run exploration 2
Export to file

#include <MyExplorations.afl>

if ( NOT StrFind(Name(), "~") ) {

    /*
        Backtest code with custom equity symbol(s) creation
    */

}

if ( Name() == "~~~Equity1" )
    RunExploration1();

if ( Name() == "~~~Equity2" )
    RunExploration2();

fxshrat · February 7, 2022, 10:36am

fxshrat:

Batch (portfolio backtest):

Load apx
Backtest
Run exploration 1
Export to file
Run exploration 2
Export to file

#include <MyExplorations.afl>

if ( NOT StrFind(Name(), "~") ) {

    /*
        Backtest code with custom equity symbol(s) creation
    */

}

if ( Name() == "~~~Equity1" )
    RunExploration1();

if ( Name() == "~~~Equity2" )
    RunExploration2();

Variation:

Batch (portfolio backtest):

Load apx
Backtest
Run exploration 1
Export to file
Scan to CategoryAdd/Remove
Run exploration 2
Export to file

#include <MyExplorations.afl>

watchlist_number = GetOption("FilterIncludeWatchlist");
is_scan = Status( "action" ) == actionScan;

// batch step - Backtest
if ( NOT StrFind(Name(), "~") AND NOT is_scan ) {

    /*
        Backtest code with custom equity symbol(s) creation
    */

    // within CBT Equity symbol creation
    CategoryAddSymbol("~~~Equity1", categoryWatchlist, watchlist_number); 
    CategoryRemoveSymbol("~~~Equity2", categoryWatchlist, watchlist_number);  
}

// First Exploration batch step
if ( Name() == "~~~Equity1" ) {
    RunExploration1();     
}

// Batch step - Scan
if ( is_scan AND Status("Stocknum") == 0 ) {
    CategoryRemoveSymbol("~~~Equity1", categoryWatchlist, watchlist_number);  
    CategoryAddSymbol("~~~Equity2", categoryWatchlist, watchlist_number);
}

// 2nd Exploration batch step
if ( Name() == "~~~Equity2" ) {
    RunExploration2();      
}

fxshrat · February 7, 2022, 2:22pm

Here is working code.
I tested it and it works!

Your first exploration is based on AddRow and your second one on AddColumn only.

//// @link https://forum.amibroker.com/t/efficiency-in-the-automation-of-tasks/29623/8
/// Batch (portfolio backtest):
///    Load apx
///    Backtest
///    Run exploration 1
///    Export to file
///    Scan to CategoryAdd/Remove
///    Run exploration 2
///    Export to file 
    
//#include <MyExplorations.afl>

// Put functions to include file
procedure RunExploration1(nm) {
	global Filter;	
	SetOption("NoDefaultColumns", 1);
	Filter = 0;
	AddColumn(Null, "Explore Equity1", 1.2);
	if ( StrToUpper(Name()) == nm )
		AddRow(StrFormat("%g", LastValue(C)));
}

procedure RunExploration2(nm) {
	global Filter;	
	Filter = StrToUpper(Name()) == nm;
	AddColumn(C, "Explore Equity2", 1.2);
}

procedure RunExploration(nm) {
	global Filter;	
	// based on AddRow
	if ( nm == "~EQUITY1" ) {		
		RunExploration1(nm);
	}	
	// based on AddColumn only
	if ( nm == "~EQUITY2" ) {
		RunExploration2(nm);
	}
}

watchlist_number = GetOption("FilterIncludeWatchlist");
is_scan = Status( "action" ) == actionScan;

// batch step - Backtest
if ( NOT StrFind(Name(), "~") AND NOT is_scan ) {
    /*
        Backtest code
    */    
    period = 20; // number of averaging periods 
	m = MA( Close, period ); // simple moving average
	Buy = Cross( Close, m ); // buy when close crosses ABOVE moving average
	Sell = Cross( m, Close ); // sell when closes crosses BELOW moving average
	Short = Cover = 0; 
}

// custom equity symbol(s) creation
SetCustomBacktestProc("");
if( Status("action") == actionPortfolio )
{
	bo = GetBacktesterObject();
	bo.Backtest();
	AddToComposite( bo.EquityArray,	"~Equity1", "X",atcFlagDeleteValues | atcFlagEnableInPortfolio );
	AddToComposite( bo.EquityArray,	"~Equity2", "X",atcFlagDeleteValues | atcFlagEnableInPortfolio );
	
	CategoryAddSymbol("~Equity1", categoryWatchlist, watchlist_number); 
	CategoryRemoveSymbol("~Equity2", categoryWatchlist, watchlist_number); 
}

// Batch step - Scan
if ( is_scan AND Status("stocknum") == 0 ) {
    CategoryRemoveSymbol("~Equity1", categoryWatchlist, watchlist_number);  
    CategoryAddSymbol("~Equity2", categoryWatchlist, watchlist_number);
}

// Exploration batch step(s)
sym_list = CategoryGetSymbols(categoryWatchlist, watchlist_number);// all upper case
sym_num = StrCount(sym_list, ",");
last_sym = StrExtract(sym_list, sym_num);
_TRACE(last_sym);

RunExploration(last_sym);

Vegeta · February 7, 2022, 3:30pm

Almost...

I think there is something not compatible with the explorers that I run.

I put the if like that to try to avoid execute the exploration for each symbol.

	if ( StrToUpper(Name()) == nm )
		#include <MyExplorations_Equity.afl>

//// @link https://forum.amibroker.com/t/efficiency-in-the-automation-of-tasks/29623/8
/// Batch (portfolio backtest):
///    Load apx
///    Backtest
///    Run exploration 1
///    Export to file
///    Scan to CategoryAdd/Remove
///    Run exploration 2
///    Export to file 
    
// Put functions to include file
procedure RunExploration1(nm) {
	global Filter;
	Filter = 0;
	if ( StrToUpper(Name()) == nm ){
		#include <MyExplorations_Equity.afl>
	}
}

procedure RunExploration2(nm) {
	global Filter;
	Filter = 0;
	if ( StrToUpper(Name()) == nm ){
		#include <MyExplorations_TableProfit.afl>
	}

}

procedure RunExploration(nm) {
	global Filter;	
	// based on AddRow
	
	
	if ( nm == "~EQUITY1" ) {		
		RunExploration1(nm);
	}	
	// based on AddColumn only
	if ( nm == "~EQUITY2" ) {
		RunExploration2(nm);
	}
}

watchlist_number = GetOption("FilterIncludeWatchlist");
is_scan = Status( "action" ) == actionScan;

// batch step - Backtest
if ( NOT StrFind(Name(), "~") AND NOT is_scan ) {
    /*
        Backtest code
    */    
    period = 20; // number of averaging periods 
	m = MA( Close, period ); // simple moving average
	Buy = Cross( Close, m ); // buy when close crosses ABOVE moving average
	Sell = Cross( m, Close ); // sell when closes crosses BELOW moving average
	Short = Cover = 0; 

}
// custom equity symbol(s) creation
SetCustomBacktestProc("");
if( Status("action") == actionPortfolio )
{
	bo = GetBacktesterObject();
	bo.Backtest();
	AddToComposite( bo.EquityArray,	"~Equity1", "X",atcFlagDeleteValues | atcFlagEnableInPortfolio );
	AddToComposite( bo.EquityArray,	"~Equity2", "X",atcFlagDeleteValues | atcFlagEnableInPortfolio );
	
	CategoryAddSymbol("~Equity1", categoryWatchlist, watchlist_number); 
	CategoryRemoveSymbol("~Equity2", categoryWatchlist, watchlist_number); 
}

// Batch step - Scan
if ( is_scan AND Status("stocknum") == 0 ) {
    CategoryRemoveSymbol("~Equity1", categoryWatchlist, watchlist_number);  
    CategoryAddSymbol("~Equity2", categoryWatchlist, watchlist_number);
}

// Exploration batch step(s)
sym_list = CategoryGetSymbols(categoryWatchlist, watchlist_number);// all upper case
sym_num = StrCount(sym_list, ",");
last_sym = StrExtract(sym_list, sym_num);
_TRACE(last_sym);

RunExploration(last_sym);

#include <MyExplorations_Equity.afl>


fbr = Status("firstbarinrange");// flag first bar of any analysis range
firstClose = LastValue(ValueWhen(Ref(fbr,-2), Close));// firstClose - not an array b

Filter = 1;
//eqname = "~~~EQUITY";
eqname=ParamStr("Ticker","~~~XXXX");

if( Name() != eqname ) SetForeign( eqname );
eq = Close;
Cash = Low;

dr = eq - Highest(eq);

// BENCHMARK
benchmark="^SPX";
SetForeign(benchmark);
eq2 = Close;
Cash2 = Low;
dr2 = eq2 - Highest(eq2);

fbr2 = Status("firstbarinrange");// flag first bar of any analysis range
firstClose2 = LastValue(ValueWhen(Ref(fbr,-1), Close));// firstClose - not an array b

RestorePriceArrays();
InitialEquity= firstClose;
InitialEquity2=firstClose2;

SetOption( "NoDefaultColumns", True );

AddColumn( DateTime(), "Date/Time", formatDateTime);

AddColumn(firstClose, "firstClose");

AddColumn(eq, "Portfolio Equity");
AddColumn(ROC(C,1), "Portfolio Equity PTC");
AddColumn(cash, "Cash");
AddColumn(dr, "Drawdown");
//AddColumn((eq/cash)*100, "Drawdown %"); // nueva
AddColumn((dr/Highest(eq))*100, "Drawdown %"); // nueva
AddColumn((eq/100000-1)*100, "Rentabilidad %"); // nueva

AddTextColumn(Name(),"Symbol");

AddColumn(eq2, "Benchmark");
AddColumn(ROC(eq2,1), "Benchmark PTC");
AddColumn(cash2, "Benchmark Cash");
AddColumn(dr2, "Benchmark Drawdown"); 
AddColumn((dr2/Highest(eq2))*100, "Benchmark Drawdown %"); // nueva
AddColumn((eq2/InitialEquity2-1)*100, "Rentabilidad Benchmark %"); // nueva

AddTextColumn(benchmark,"Benchmark Symbol");

#include <MyExplorations_TableProfit.afl>

/*
   MonthlyTable_01.afl

   Exploration code to produce a month-by-month tabular presentation of percentage changes
   in a value.

   In this code the value is designated as "eq" (for equity), but is simply set to the 
   price Close for demonstration purposes.
   
   Original code provided by "Mike", here:

      http://finance.groups.yahoo.com/group/amibroker/message/125846

   Extra comments, _TRACE, and some changes by Progster

*/


//   Calculation of the value for which the tabular display of %Chg is desired.
//   Change this to whatever calculation you wish.
//SetForeign("~~~BenderDiario2022_SinDividendo");
eq = C;

yr = Year();
mo = Month();

YearChange = yr != Ref( yr, 1 );
MonChange = mo != Ref( mo, 1 );

FirstYr = 0;
LastYr = 0;

////////////////////////////
// SKIP non-trading bars
////////////////////////////
// Removed loop
bi = BarIndex();
fbr = Status("firstbarinrange");
//is_equal = SumSince(fbr,eq==0) == BarsSince(fbr);
is_equal = SumSince(fbr,Nz(eq)==0) == BarsSince(fbr); // cambio fxshrat
startbar = LastValue(ValueWhen(is_equal, bi));
endbar = BarCount - 1;

////////////////////////////
// collect yearly / monthly changes in equity
// into dynamic variables
////////////////////////////

LastYrValue = eq[ startbar ];
LastMoValue = eq[ startbar ];
MaxYrProfit = MinYrProfit = 0;
MaxMoProfit = MinMoProfit = 0;

for ( i = startbar + 1; i <= endbar; i++ )
{
    if ( YearChange[ i ] || i == endbar )
    {
        Chg = 100 * ( -1 + eq[ i ] / LastYrValue );
        VarSet( "ChgYear" + yr[ i ], Chg );

        MaxYrProfit = Max( MaxYrProfit, Chg );
        MinYrProfit = Min( MinYrProfit, Chg );

        if ( FirstYr == 0 )
            FirstYr = yr[ i ];

        LastYr = yr[ i ];

        LastYrValue = eq[ i ];
    }

    if ( MonChange [ i ] || i == endbar )
    {
        mon = mo[ i ];

        Chg = 100 * ( -1 + eq[ i ] / LastMoValue );

        VarSet( "ChgMon" + yr[ i ] + "_" + mon, Chg );
        VarSet( "SumChgMon" + mon, Chg + Nz( VarGet( "SumChgMon" + mon ) ) );
        VarSet( "SumMon" + mon, 1 + Nz( VarGet( "SumMon" + mon ) ) );

        MaxMoProfit = Max( MaxMoProfit, Chg );
        MinMoProfit = Min( MinMoProfit, Chg );

        LastMoValue = eq[ i ];
    }
}

//MonthNames = "Jan,Feb,Mar,Apr,May,Jun,Jul,Aug,Sep,Oct,Nov,Dec";
MonthNames = "Ene,Feb,Mar,Abr,May,Jun,Jul,Ago,Set,Oct,Nov,Dic";

if ( Name() != "~~~EQUITY" AND Name() != "~~~OSEQUITY" )
{
	printf( "For accurate results switch to ~~~EQUITY symbol<br>" );
}

////////////////////////////
// Main program
////////////////////////////
SetOption("NoDefaultColumns", 1);
Filter = 0;
SetSortColumns( -1 ); 

printf( "<table border='1' bordercolor='#000000' cellspacing='0' cellpadding='3'style='border-collapse:collapse;'>\n" );
printf( "<tr bgcolor='#eeffee' >\n" );

Header = "Año," + MonthNames + ",Anual%%";

for ( Col = 0; ( Colname = StrExtract( Header, Col ) ) != ""; Col++ )
{
	printf( "<td><b>" + Colname + "</b></td>" );
}

printf( "</tr>\n" );

//AddColumn(Null, "Ticker", 1);
AddColumn(Null, "Año", 1);
for ( m = 1; m <= 12; m++ )
	AddColumn(Null, StrExtract(MonthNames,m-1), 1.2);
AddColumn(Null, "Anual%", 1.2);

fmt = ".2f";

for ( y = FirstYr; y <= LastYr; y++ )
{
	//Color =  ColorRGB( IIf( row == 0 || col == 0 || col == 13, 220, 255 ), 255, IIf( row % 2, 255, 220 ) );

	// new row
	if ( y % 2 )
		printf( "<tr bgcolor='#ffffff'>\n<td bgcolor='#eeffff'>" );
	else
		printf( "<tr bgcolor='#ffffee'>\n<td bgcolor='#eeffee'>" );

	printf( "<b>%g</b></td>", y );

	ar_str = "";
	for ( m = 1; m <= 12; m++ )
	{
		Chg = VarGet( "ChgMon" + y + "_" + m );

		if ( NOT IsNull( Chg ) )
		{
			if ( Chg >= 0 )
				printf( "<td nowrap>%"+fmt+"%%</td>", Chg );
			else
				printf( "<td nowrap><font color='880000'>%"+fmt+"%%</font></td>", Chg );
			
			ar_str += StrFormat("%"+fmt+"%%\t", Chg);
		}
		else {
			printf( "<td></td>" );
			ar_str += "\t";
		}
	}

	if ( y % 2 )
		//printf( "<td nowrap bgcolor='#eeffff'>" );
	//else
		printf( "<td nowrap bgcolor='#eeffee'>" );

	x = VarGet( "ChgYear" + y );

	if ( x >= 0 )
		printf( "<b>%"+fmt+"%%</b></td>", x );
	else
		printf( "<font color='880000'><b>%"+fmt+"%%</b></font></td>", x );
	
	x_str = WriteIf(IsNull(x), "N/A", StrFormat("%"+fmt+"%%",x));         	// fxshrat
	//AddRow(StrFormat("%s\t%g\t%s%"+fmt+"%%",Name(),y,ar_str,x));   			// fxshrat        
	AddRow(StrFormat("%g\t%s%"+fmt+"%%",y,ar_str,x));  // no ticker column

	printf( "</tr>\n" ); // end row
}


printf( "<tr bgcolor='#eeffee' >\n" ); // new row
printf( "<td><b>Avg</b></td>" );

ar_str = "";
for ( m = 1; m <= 12; m++ )
{
	x = Nz( VarGet( "SumChgMon" + m ) / VarGet( "SumMon" + m ) );

	if ( x >= 0 )
		printf( "<td nowrap><b>%"+fmt+"%%</b></td>", x );
	else
		printf( "<td nowrap><font color='880000'><b>%"+fmt+"%%</b></font></td>", x );
	
	ar_str += StrFormat("%"+fmt+"%%\t", x);
}
//AddRow(Name()+"\tAverage:\t"+ar_str);

printf( "<td>&nbsp;</td>" );
printf( "</tr></table>\n" );

If you try to create the two includes / explorers and run it only on the current symbol example over. ~Equity1 you will see that it works fine, but inside the AFL that we are trying to set up its behavior is completely different and I don't know why

Thank you very much

Vegeta · February 7, 2022, 3:34pm

But there are factors that I think make it impossible for us to do it that way, for example

the backtest has to be run with pad&align and daily
and instead the explorer of the profit table must be run without pad&align and in monthly for it to work well.

I don't think that can be fixed, right?

Tomasz · February 7, 2022, 3:38pm

There is nothing to "fix", except invalid assumptions.
Pad and align is controllable from settings. And settings are part of Analysis project file. You can have 2 projects one with pad and align turned on and one with it turned off.

Vegeta · February 7, 2022, 3:46pm

Hi,

Maybe I need an APX for each backtest
I generate a batch that executes them all, with that I already have the curves of each one

And then maybe with just 1 APX with the explorer in current symbol and call it from the batch for each system, putting the set current ticker with the name of each system equity

And the same for the profit table explorer but with another configuration of settings without pad&align and monthly, putting the set current ticker with the name of each system equity.

My summary...
1APX per system
1 batch for all systems

1 APX explorer equity curve
1 batch for all equity curves

1 APX explorer table profit
1 batch for all table profit

I can't find any alternative that prevents me from creating less APX or batch.

Right?

Tomasz · February 7, 2022, 3:48pm

No, you don't. @fxshrat explained it to you several times. Re-read his responses carefully.

Vegeta · February 7, 2022, 4:00pm

Hi,

Yes, I have read and reread believe me hehe

If I understood well, the form of @fxshrat works when you don't have to change preferences, (but in my scenario it fails me with my includes).

But if for the backtest I need daily and pad&align and for one of the two explorers that is in monthly and without pad&align as you say, there is no other option but to have 2 APX that work, and in the best scenario call a batch and those 2 apx for each strategy. This is true that it would only have one batch instead of 3 and less APX...only 2 per backtest.

If I haven't said anything wrong this time

fxshrat · February 7, 2022, 4:28pm

Not correct. Look at gif animation and code.

Vegeta:

// Put functions to include file
procedure RunExploration1(nm) {
	global Filter;
	Filter = 0;
	if ( StrToUpper(Name()) == nm ){
		#include <MyExplorations_Equity.afl>
	}
}

procedure RunExploration2(nm) {
	global Filter;
	Filter = 0;
	if ( StrToUpper(Name()) == nm ){
		#include <MyExplorations_TableProfit.afl>
	}

}

You are doing non sense here.
You set Filter to zero two times.

Also you should set Filter and AddColumns as I did but not within If statement checking for symbol (it is within if statement on your end because you include everything of your exploration without thinking).

You just copy and paste instead of understanding!

If you need monthly and do not want to use additional APX then use timeframe functions.
(But AFAICS, the outcome should be the same with daily TF since it is monthly rate of change table already.)

And why putting pad&align off?

system · May 18, 2022, 4:29pm

This topic was automatically closed 100 days after the last reply. New replies are no longer allowed.