Sell = 1 not sending a sell signal to Custom Backtester every day

Hello,

As a quick precursor, I have done both a google search and search in this forum, but couldn't find a similar post dealing with this issue. I have also read the documentation on how buy and sell arrays and ExRem works. If there is existing information then I do apologize, please just provide a link for me, and I will happily go and read it myself.

I've been writing a rotational system with a market gate that uses the low level custom backtester (CBT). I do use the Buy signals, but since my exit logic is contained entirely in the CBT and I had no need for Sell signals, I used the Sell signal to encode whether or not this was a rotation day or whether the market gate had shut. Essentially this worked by setting Sell = 1 (outside of CBT) on days when trading/rotation should take place. Then inside the CBT, I would use code similar to the following:


		for ( sig = bo.GetFirstSignal( bar ); sig; sig = bo.GetNextSignal( bar ) )	//	loop through all signals
		{
			if( sig.IsExit() )	//	if we get an sell signal, we must have set Sell = 1
			{
				ExitSignalNum++; 
				if (ExitSignalNum > 0) 
				{
					_TRACE( StrDate + ": and we have at least 1 Sell signal over the watchlist." );
					break;	
				}
			}
		}

This would determine if I was getting Sell signals, which meant this is a rotation/trading day. Recently I have realized that some days that should be marked as trading days are not trading during backtesting. In an effort to debug, I set an unrestricted Sell = 1, which to my understanding should mean that for every bar I backtest over, there should be a sell signal for every stock in the basket I am backtesting over. I have not used ExRem, so I don't see why any signals should be removed.

I have then used TRACE and the window log, and I see that there are some bars which don't have sell signals.

image

If you can see the screenshot, it shows that on the 28 September, we do iterate through that bar, but we do not get a sell signal for some reason. The days before it seem ok.

This is difficult for me to understand. To further troubleshoot, when I set Buy = 1, I see that in this case I do have a buy signal for each bar. So I am assuming that it has something to do with one being an entry signal and the other being an exit signal?

If this is the case, since I need the real Buy signal in my logic, I can't very well use the Buy signal to encode whether it is or is not a trading day, but I could possibly use the Short signal. However, I would prefer to understand what is causing the Sell signal not to be transmitted every bar to the CBT, instead of just using a workaround.

If anyone could explain what is going on to me, or refer me to further information, I would be most grateful.

Please read this page to see how AmiBroker filters your entry and exit signals based on the Backtester Mode https://www.amibroker.com/guide/h_portfolio.html

Since your rotation day is probably the same for all symbols, you could simply set a Static Variable in your Phase 1 code and retrieve it during Phase 2 (CBT). The static variable would be True for each bar that's a rotation bar.

1 Like

Take a look at:
https://www.amibroker.com/guide/afl/setbacktestmode.html

1 Like

And obviously rotational system should use rotational backtest mode.

Thank you all for the replies. Seeing that I should have the backtest mode set correctly was like a Eureka moment after spending hours troubleshooting other areas. When I built the system years ago, I was using backtestRegularRaw2 which would keep these redundant sell signals that I needed. In all my fiddling and meddling with the code I must have deleted the "2" somewhere along the way.

Was also a good tip from mradtke of using a static variable instead (setting in Phase 1 and retrieving in phase 2), this would probably be more efficient. I'm not even sure why I didn't think of this myself when I built the system, as I use static variables all the time.

Anyway, in a few lines from the community, you've all managed to solve a problem that had me stumped for ages. Very grateful to all!

1 Like

You can also express your gratitude with the Like icon or by marking a post as the Solution to your question. :slight_smile:

1 Like

Done, apologies for not using correct protocol.

In trying to fix this code, I do remember why I didn't use a static variable now. I use a market gate with a 200 day MA crossover on the S&P 500 index. If I use a static variable, I don't believe I get access to the time before the backtest to create the MA, hence I have to start the backtest some 200 days before I can start to get market gate signals, making the backtests inaccurate.

When using Sell signals to encode from phase 1 to phase 2, I get access to data before the backtest period to allow me to have my market gate available from day 1.

That being said even when changing the backtest mode from RegularRaw to RegularRaw2, this solved the earlier time periods when I was not getting sell signals transmitted, but strangely enough now gives me 2 extra trading days that shouldn't be trading days (in essence sell signals on 2 days that shouldn't have sell signals). When I encode "CanTrade" array to a static variable array and encode "CanTrade" array to a Sell array, for some reason the static variable works perfectly but the Sell array gives me 2 extra "CanTrade" days, even with backtest mode RegularRaw2. Can you think of any reason it might be doing this?

Please disregard paragraph 3, for some reason when I restart AB everything seems to work again. I'm not sure if Sell arrays are kept from previous backtest runs or something like that, but I'm just grateful it's working. I tried to edit post, but apparently it was posted too long ago to edit or delete.

The first 2 paragraphs explaining why I encoded in Sell array instead of static variable are still valid.

Thanks.

Static Variables if not Set Persist=True will be destroyed in a Restart.
so each restart gives you a clean slate in that respect.

If you are not resetting static vars properly, you maybe having stale data or something similar.

1 Like

In Phase 1, AmiBroker will attempt to give you enough bars prior to the start of your analysis date range so that you can accurately calculate all indicators. If you use Foreign/SetForeign, then that data will be aligned with the current symbol as well. If you have Pad & Align enabled in the Analysis Settings, then all data will be aligned with the alignment symbol. Therefore, in Phase 1 you should be able to calculate MA(200) of the market symbol and save it into a Static Variable if desired.

In Phase 2, Bar 0 will generally correspond to the first date of the analysis date range, which means that you cannot calculate MA(200) until 200 bars into the backtest. That's why you need to do the calculation in Phase 1.

Are you trying to store your "CanTrade" Static Variable just once, or once per symbol?

1 Like

Thanks for the replies.

I haven't been setting Persist=True (default is false), so I should be getting a clean slate each time.

I have been using Foreign in phase 1. I had not been using Pad & Align, but I tried it again with pad and align to the S&P 500 index (which is my market symbol) and it did not make a difference to whether I get an erroneous backtest or not. In phase 1 I use:

StaticVarSet("CanTradeStaticVarArray", IIf( TradingAllowed, 1, 0 ));

So I assume that does it once per symbol. But while that seems inefficient, I don't see why that would be erroneous. I then use StaticVarGet in phase 2 (CBT) and then take the bar number of the array to check whether that bar should be a trading bar or not. When the backtest is long enough to give it enough data for a moving average, it doesn't create errors, but when the backtest is too short, or it is too early in the backtest to get the MA, then the bars aren't recognized as trading bars.

Thanks for the responses. I have managed to get this working using Sell signals, so to fix the code using Static variables would be an academic exercise only. Please note, I'm happy to see that academic exercise to completion, if there is interest, as I feel it is the least I can do for all the help I've received. But I've been able to get a result using Sell signals, so if everyone has more pressing issues, please feel free to let this be the closing post.

There are various examples on this little bit of code which is used to ensure that a block of code is run only once which i'm sure you know of. Especially for Ranking etc.

if ( Status("stocknum") == 0 ) { /* ...some run once code */ }
You can use the Static Var and compute it once.

1 Like

No, it is not per symbol.
As for per symbol static var you would have to add Name() function.

StaticVarSet("CanTradeStaticVarArray_" + Name(), IIf( TradingAllowed, 1, 0 ));

Then e.g. for calling array in CBT

can_trade = StaticVarGet("CanTradeStaticVarArray_" + sig.Symbol);

my_other_code = ... can_trade[ i ]...;

As aside

Using static variables to pass variables from 1st phase to 2nd phase is not new invention but it is (very) old and is known for long time. It was introduced by AmiBroker by AmiBroker developer years of years ago but by no one else.

For example this is from 2008:

https://groups.yahoo.com/neo/groups/amibroker/conversations/messages/119901

So even Neanderthals were applying it that way during ice age already (since AmiBroker have invented time machine also). It is not an invention or new idea by any user not to mention by any scam artists trying to suck in people to thousands of Dollars courses about "new secret" information that are not secret at all but that have been publicly available for years. There isn't anything new under the sun and there isn't almost any secret out there. Almost everything is stone cold old basics and just repetition. If you see such infos to be sold as super confidential secret and new invention for large amount of money then you can be rest assured it is crystal clear scam but nothing else.

Old stuff but not new.

1 Like

Hello fxshrat. It's good to see that you are still here helping people as I've been away from Amibroker (and hence AB forum) for a long time, and I remember you from years ago. Always incredibly insightful with an incredibly deep understanding of Amibroker, and I always appreciated your input, however blunt and to the point it was, hahaha.

I'll address the

Blockquote

quote made by myself. I realized that it did not create a separate trading 'gate' for each security in the S&P 500 watchlist I was running it on. This is all I needed, a general market gate for all securities, so no need to set a gate for each security. But I believed that for each security in the watchlist, I called the foreign "market" security, then set a static variable based on it:

Trigger1 = Foreign("$SP","C");
Gate1 = MA( Trigger1, 253 );	//	253 trading days is 1 year
StaticVarSet("CanTradeStaticVarArray", IIf( Gate1, 1, 0 ));

Hence I think I would set the same static variable some 500 times, once for each security in my watchlist (I might be wrong in my understanding). Which was very inefficient I guess. Travick's suggestion of using:

if ( Status("stocknum") == 0 )

would get rid of this inefficiency, so I believe it would be a good suggestion if I was to pursue that route (which is not required since I managed to get it working using the Sell signal).

In terms of your statement about whether the use of static variables and passing from phase 1 to phase 2 not being a new invention, but as old as time and Amibroker itself, I'm not sure if it was something I said to incur your wrath, or is an old argument with another poster. In my defense, I said:

Blockquote

So I did say it was a tip and by that I meant hint, reminder if you will. Not that it was an invention or new idea as such. I also have no intention of spending money on any super confidential secret, as like you say, the only super confidential secret is that Amibroker is better and available more cheaply than any of the more expensive, less effective competitor software packages.

Anyway, since I have managed to get my code working and I don't want to cause any arguments amongst anyone, I'll say thank you to all that have contributed hints to help me and just say that I truly do appreciate people that go out of their way to help others with their AB programming troubles. Thanks again all.

1 Like

No, you misunderstood. I wasn't referring to you but I meant that I have heard stories already over several years that known publicly available information have been sold as super confidential new information. And I am quite certain there still exist such people. So it was general comment that there isn't almost anything new under the sun.

1 Like

BTW

Do you want to check whether MA is NULL?

// Gate1 returns 0 or 1 so no IIf() required
Gate1 = ! IsNull(MA( Trigger1, 253 )); // ! means "NOT"

Or do you want to check whether price is above/below MA?

Trigger1 = Foreign("$SP","C"); 
// bullish market if C > MA, returns 0 or 1 so IIf() not required
Gate1 = Nz(C > MA( Trigger1, 253 )); // 253 trading days is 1 year 
1 Like
SetCustomBacktestProc( "" );
Buy  = Sell = Short = Cover = 0;

if( Status( "action" ) == actionPortfolio )
{
    bo = GetBacktesterObject();	//  Get backtester object
    bo.PreProcess();	//  Do pre-processing (always required)


    SetForeign( "Your Symbol" )  ;
    gate = Close > MA( Close, 200 );
    RestorePriceArrays() ;

    StaticVarSet( "gate", gate ) ;

    for( i = 0; i < BarCount; i++ )	//  Loop through all bars
    {
        for( sig = bo.getfirstsignal();  sig ; sig = bo.GetNextSignal() )
        {
            if( ! gate[i] )
                sig.Price = -1;     // gate closed : ignore signal
        }

        bo.ProcessTradeSignals( i );	//  Process trades at bar (always required)
    }

    bo.PostProcess();	//  Do post-processing (always required)
}


// plot results
gate = StaticVarGet( "gate" ) ;
color = IIf( gate  , colorGreen , colorRed ) ;
color = IIf( Status( "barinrange" ) , color , colorDarkGrey ) ;
Plot( C, "", color , styleBar ) ;

1 Like

@aron, your solution won't work in this case for reasons discussed above.

@Narcissus, one subtlety that wasn't mentioned previously is that your Phase 1 code is executed once for every symbol in the watchlist and then once more with the symbol ~~~EQUITY. The equity symbol does not have additional history from before the start of the backtest. Since this is your final time through the Phase 1 code and therefore your final time setting your static variable, the value used by the CBT is the one that didn't have enough history to calculate the MA(200). You could resolve this by using the Status("stocknum") == 0 logic that @travick referenced. Sorry I didn't highlight the ~~~EQUITY issue sooner.

2 Likes

I do not see why.
All it takes is :

  1. set the start date earlier so that market average can be calculated
  2. Ignore all trading signals if market average is null with a slight modification
for( sig = bo.GetFirstSignal( i ); sig; sig = bo.GetNextSignal( i ) )
{
    if( gate[i] )
    {
    }
    else
    {
        sig.Price = -1;
    }
}