Weird Difference between Explore & Backtest

I have run into an oddity between Explore & Backtest.

  • I'm backtesting a futures system

  • I'm running just Short Trades only
    image

  • Here is a screenshot of the chart that I'm having a problem with.

  • The red painted bar shows that the conditions for that bar and the prior bar indicate a Short is to be taken the next day, if possible.

  • Please note that the setup and trade illustrated by the first down arrow and the up arrow are correct.

  • I cannot account for the second down arrow.
    image

  • Here is a combined screenshot of the Exploration and the log

  • Notice that log window is sorted in ascending order.

  • Notice that there is no Short entry on 1999-02-11
    image

  • I cleared the Log and the Exploration.

  • I ran Backtest

  • Here is a screenshot of the Backtest, the Report, and the Log.
    image

Other things to note:

  • Using breakpoint and stepping, I stepped through the data.
  • I setup a watchlist and looked at the variables.
  • At no point did a second Short entry occur on 1999-02-11

Just in case you need it, here is a screenshot of the SetOptions at the beginning of my code:
image

I am at a complete loss as to how to find this problem. Can anyone give me a clue of where to look?

Thanks.

Exploration is NOT backtest. It was explained dozens of times already:

https://forum.amibroker.com/search?q=exploration%20is%20not%20backtest

Exploration serves completely different purpose than Backtest and uses DIFFERENT set of functions than Backtest. Those two modes are separate. Backtest-only functions like SetTradeDelays, ApplyStop, many SetOption calls have absolutely NO effect on exploration.

Also code should be included as code (text), not as picture. Please follow this advice: How to ask a good question

To debug backtests, use Detailed log as explained here: How do I debug my formula? as this provides a whole lot more information than exploration.

Tomasz,

Insulting a user who paid you good money for your program is NOT a good practice, and certainly NOT good customer relations. If your documentation were clearer, then you wouldn't be beset with these questions. For example, you have jumbled Futures settings together with Stocks settings and it is very hard to determine what applies to each. Here's another example: You use the words "Detailed Log" to refer to the result window, but it does not refer to the Log Window. That's confusing. So, before you go insulting people who give you money, you might want to take a look in the mirror first.

  • I did indeed ask a good question.
  • The reason I did not include code is because
    (1) an obvious error might have been clear from my detailed description of the things that I did,
    including detailed Exploration, _Trace, and Stepping through code
    (2) I did not want to make the post inordinately long
    (3) !f I was incorrect that it was simple, then a simple answer of "We are unable to tell without
    the code" would have been well-received and I would have followed up with the code.

The "Detailed Log" told me nothing:
image

So, here is the code:

// Backtest settings
Filter = InWatchListName( "BackAdjusted" );
SetPositionSize( 1, spsShares );
SetBacktestMode( backtestRegularRaw );
SetOption( "InitialEquity", 50000 );
SetOption( "AllowSameBarExit", True );
SetOption( "ActivateStopsImmediately", True );
SetOption( "FuturesMode", True );
SetOption( "MaxOpenPositions", 1 );
SetOption( "ReverseSignalForcesExit", True );
//SetOption( "SeparateLongShortRank",True );
//SetOption( "MaxOpenLong", 2 );
//SetOption( "MaxOpenShort", 2 );
SetOption( "AccountMargin", 100 );
SetOption( "CommissionMode", 2 );
setOption( "CommissionAmount", 0 );
SetOption( "EveryBarNullCheck", True );
SetOption( "DisableRuinStop", True );
//SetOption( "PriceBoundChecking", True );

// Common Calculations
midpt = ( H + L ) / 2;							// bar midpt
h1 = Ref( H, -1 );								// Yesterday's High
l1 = Ref( L, -1 );								// Yesterday's Low
c1 = Ref( C, -1 );								// Yesterday's Close
c2 = Ref( C, -2 );								// Close 2 days ago
th = Max( High, Ref( Close, -1 ) );				// True High
tl = Min( Low, Ref( Close, -1 ) );				// True Low
th1 = Ref( th, -1 );								// Yesterday's True High
tl1 = Ref( tl, -1 );								// Yesterday's True Low
tr = th - tl;									// True Range
tr1 = Ref( tr, -1 );							// Yesterday's True Range
df = MA( C, 3 );								// Directional Filter
df1 = Ref( df, -1 );							// Directional Filter one day ago
df2 = Ref( df, -2 );							// Directional Filter two days ago
tf = df2;										// Trade Filter (Filters out normal signal)
trnside = ( H <= th1 ) && ( L >= tl1 );				// True Inside Day

// Two Different Entry Points
rawbb = c1 + tr1;								// Raw Buy Band, i.e. Normal Signal
rawsb = c1 - tr1;								// Raw Sell Band, i.e. Normal Signal
bb = Max( tf, rawbb );							// Buy Band: Max of Normal Signal and TF
sb = Min( tf, rawsb );							// Short Band: Min of Normal Signal and TF

// Long Conditions
bcnd1 = c1 > df2;
bcnd2 = ( L > sb ) && ( h1 < Ref( bb, -1 ) );
bcnd3 = l < min( tf, df1 );
bcnd4 = C > df1;
bcnd5 = NOT( trnside || Ref( trnside, -1 ) );
bcnd6 = H < bb;
bcnd7 = C > O;
bcnd = 	bcnd1 &&
        bcnd2 &&
        bcnd3 &&
        bcnd4 &&
        bcnd5 &&
        bcnd6 && bcnd7;

// Short Conditions
scnd1 = c1 < df2;
scnd2 = ( H < bb ) && ( L1 > Ref( sb, -1 ) );
scnd3 = H > Max( tf, df1 );
scnd4 = C < df1;
scnd5 = NOT( trnside || Ref( trnside, -1 ) );
scnd6 = L > sb;
scnd7 = O > C;
scnd = 	scnd1 &&
        scnd2 &&
        scnd3 &&
        scnd4 &&
        scnd5 &&
        scnd6 && scnd7;

by = Ref( bcnd, -1 ) && H >= bb;
sl = Ref( scnd, -1 ) && L <= sb;

// Common Calculations for determining whether High or Low occurred first
r = H - L;										// Daily Range
rtop3 = H - r / 3;								// Value beginning the top third of Range
rbot3 = r / 3 + L;								// Value beginning the bottom third of Range0
rtop25 = H - ( 0.25 * r );						// Value beginning the top 25% of Range
rbot25 = L + ( 0.25 * r );						// Value beginning the bottom 25% of Range
topo = O == H;									// High occurred first because O = H; this is strongest case
boto = O == L;									// Low occurred first because O = L; this is strongest case
topc = C == H;									// Low occurred first because C = H; this is 2nd strongest case
botc = C == L;									// High occurred first because C = L; this is 2nd strongest case
upper1o = O >= rtop25;							// Open in upper 25% of range
upper2o = O >= rtop3;							// Open in upper third of range
upper3o = O > midpt;							// Open in upper half of range
lower1o = O <= rbot25;							// Open in lower 25% of range
lower2o = O <= rbot3;							// Open in lower third of range
lower3o = O < midpt;							// Open in lower half of range
upper1c = C >= rtop25;							// Close in upper 25% of range
upper2c = C >= rtop3;							// Close in upper third of range
upper3c = C > midpt;							// Close in upper half of range
lower1c = C <= rbot25;							// Close in lower 25% of range
lower2c = C <= rbot3;							// Close in lower third of range
lower3c = C < midpt;							// Close in lower half of range

// Conditions for which came first: High or Low
hf1 = topo;										// Strongest case: High hit first
hf2 = botc;										// 2nd strongest case: High hit first
hf3 = upper2o && lower2c;						// Strong case: High hit first
hf4 = upper1o && lower3c;						// Less strong case: High hit first.
lf1 = boto;										// Strongest case: Low hit first
lf2 = topc;										// 2nd strongest case: Low hit first
lf3 = lower2o && upper2c;						// Strong case: Low hit first
lf4 = lower1o && upper3c;						// Less strong case: Low hit first.
hf = hf1 || hf2 || hf3 || hf4;					// High First: any condition known to show High First
lf = lf1 || lf2 || lf3 || lf4;					// Low First: any condition known to show Low First
uf = NOT( hf && lf );							// Unknown First - don't know which is first

// Day of Entry Close-Only Stops
doe_bclstp = C < Min( tf, df1 );					// On Day of Entry, if close is below min of TF & DF1, exit Long
doe_sclstp = C > Max( tf, df1 );					// On Day of Entry, if close is above max of TF & DF1, exit Short

// Random determination of which came first: Profit Exit or Stop
rand_val = round( 9 * mtRandomA() );			// Random numbers
xf = rand_val <= 6;								// Profit hit first randomly 60% of the time
sf = 1 - xf;									// Stop hif first randomly 40% of the time

// Buy & Short Execution Prices
buyat1 = Max( O, bb );							// Execution Price is the Open or Buy Band, whichever is higher
shortat1 = Min( O, sb );						// Execution Price is the Open or Sell Band, whichever is lower

// Prepopulate variables for loop
lflat = True;									// Trading Long is flat, i.e., no Long trade
lexit = Null;									// Long Exit value set to Null
lstop = Null;									// Long Stop value set to Null
sflat = True;									// Trading Short is flat, i.e., no Short trade
sexit = Null;									// Short Exit value set to Null
sstop = Null;									// Short Stop value set to Null
ddd = DateNum();



// Loop through all bars

for( n = 0; n < BarCount; n++ )					
{
    if( n < 1200 )					// This IF stops _Trace from exceeding its limit, since problem occurs at beginning of data
    {
        datex = DateTimeFormat( "%Y-%m-%d", DateTimeConvert( 2, ddd[n] ) );
        bcont = False;									// Once match is made "continue to next iteration" becomes true
        scont = False;									// Once match is made "continue to next iteration" becomes true

        if( lflat && NOT By[n] )				// Long Market is flat and there is NO entry today
        {
            Buy[n] = 0;
            Sell[n] = 0;
            BuyPrice[n] = Null;
            SellPrice[n] = Null;
            _TRACEF( datex + ": No Long Entry" );
            bcont = True;
        }

        if( sflat && NOT sl[n] )				// Short Market is flat and there is NO entry today
        {
            Short[n] = 0;
            Cover[n] = 0;
            ShortPrice[n] = Null;
            coverPrice[n] = Null;
            _TRACEF( datex + ": No Short Entry" );
            scont = True;
        }

        if( lflat && By[n] && NOT bcont )						// Long Market is flat and there IS an entry today
        {
            Buy[n] = by[n];
            BuyPrice[n] = buyat1[n];

            if( NOT doe_bclstp[n] )
            {
                SellPrice[n] = Null;
                Sell[n] = 0;
                lexit = H[n];
                lstop = L[n];
                lflat = False;
                _TRACEF( datex + ": Long Entry Today at: " + BuyPrice[n] + "  and it did not exit Today" );
                bcont = True;
            }
            else
            {
                SellPrice[n] = C[n];
                Sell[n] = 1;
                lexit = Null;
                lflat = True;
                _TRACEF( datex + ": Long Entry Today & Stopped on Close at : " + SellPrice[n] );
                bcont = True;
            }
        }

        if( sflat && sl[n] && NOT scont )						// Short Market is flat and there IS an entry today
        {
            Short[n] = sl[n];
            shortPrice[n] = shortat1[n];

            if( NOT doe_sclstp[n] )
            {
                CoverPrice[n] = Null;
                Cover[n] = 0;
                sexit = L[n];
                sstop = h[n];
                sflat = False;
                _TRACEF( datex + ": Short Entry Today at: " + ShortPrice[n] + "  and it did not exit Today" );
                scont = True;
            }
            else
            {
                coverPrice[n] = C[n];
                Cover[n] = 1;
                sexit = L[n];
                sflat = True;
                _TRACEF( datex + ": Short Entry Today & Stopped on Close at : " + CoverPrice[n] );
                scont = True;
            }
        }

        // Market is in Long Trade - Process the Trade through Exit

        if( ( O[n] >= lexit ) && NOT( lflat && bcont ) )   				// Gap Open (gap in relation to Target)
        {
            Sell[n] = 1;
            SellPrice[n] = o[n];
            BuyPrice[n] = Null;
            lexit = Null;
            lflat = True;
            _TRACEF( datex + ": Long Exit on Gap Open at: " + SellPrice[n] );
            bcont = True;
        }

        if( ( H[n] >= lexit ) && ( L[n] <= lstop ) && ( NOT lflat ) && ( NOT bcont ) ) // Target & Stop both Hit - One Directly
        {
            Sell[n] = 1;
            SellPrice[n] = IIf( hf[n], lexit, IIf( lf[n], Min( lstop, O[n] ), iif( xf[n], lexit, Min( lstop, O[n] ) ) ) );
            BuyPrice[n] = Null;
            lexit = Null;
            lflat = True;
            _TRACEF( datex + ": Exit Long directly at: " + SellPrice[n] + " ;  hf = " + hf[n] + " ; lf = " + lf[n] + " ; xf = " + xf[n] );
            bcont = True;
        }

        if( ( h[n] >= lexit ) && ( NOT lflat ) && ( NOT bcont ) )					// Target Hit Directly or Indirectly
        {
            Sell[n] = 1;
            SellPrice[n] = lexit;
            BuyPrice[n] = Null;
            lexit = Null;
            lflat = True;
            _TRACEF( datex + ": Exit Long (Stop not hit) at Target at " + SellPrice[n] );
            bcont = True;
        }

        if( ( L[n] <= lstop ) && ( NOT lflat ) && ( NOT bcont ) )  					// Stop Hit Directly or Indirectly
        {
            Sell[n] = 1;
            SellPrice[n] = Min( lstop, O[n] );
            BuyPrice[n] = Null;
            lexit = Null;
            lflat = True;
            _TRACEF( datex + ": Long Stopped out (Target not hit) at: " + Sellprice[n] );
            bcont = True;
        }

        else																// No Signal
        {
            if( NOT bcont )
            {
                Sell[n] = 0;
                SellPrice[n] = Null;
                BuyPrice[n] = Null;
                _TRACEF( datex + ": No Signal" );
                bcont = True;
            }
        }

// Market is in Short Trade - Process the Trade through Exit

        if( ( O[n] <= sexit ) && ( NOT sflat ) && ( NOT scont ) )   				// Gap Open (gap in relation to Target)
        {
            Cover[n] = 1;
            coverPrice[n] = o[n];
            shortPrice[n] = Null;
            sexit = Null;
            sflat = True;
            _TRACEF( datex + ": Short Exit on Gap Open at: " + coverPrice[n] );
            scont = True;
        }

        if( ( L[n] <= sexit ) && ( H[n] >= sstop ) && ( NOT sflat ) && ( NOT scont ) ) // Target & Stop both Hit - One Directly
        {
            Cover[n] = 1;
            coverPrice[n] = IIf( lf[n], sexit, IIf( sf[n], Max( sstop, O[n] ), iif( xf[n], sexit, Max( sstop, O[n] ) ) ) );
            shortPrice[n] = Null;
            sexit = Null;
            sflat = True;
            _TRACEF( datex + ": Exit Short directly at: " + coverPrice[n] + " ;  lf = " + lf[n] + " ; hf = " + hf[n] + " ; xf = " + xf[n] );
            scont = True;
        }

        if( ( l[n] <= sexit ) && ( NOT sflat ) && ( NOT scont ) )					// Target Hit Directly or Indirectly
        {
            Cover[n] = 1;
            coverPrice[n] = sexit;
            shortPrice[n] = Null;
            sexit = Null;
            sflat = True;
            _TRACEF( datex + ": Exit Short (Stop not hit) at Target at " + coverPrice[n] );
            scont = True;
        }

        if( ( h[n] >= sstop ) && ( NOT sflat ) && ( NOT scont ) )  					// Stop Hit Directly or Indirectly
        {
            Cover[n] = 1;
            coverPrice[n] = max( sstop, O[n] );
            shortPrice[n] = Null;
            sexit = Null;
            sflat = True;
            _TRACEF( datex + ": Short Stopped out (Target not hit) at: " + Coverprice[n] );
            scont = True;
        }

        else																// No Signal
        {
            if( NOT scont )
            {
                Cover[n] = 0;
                coverPrice[n] = Null;
                shortPrice[n] = Null;
                _TRACEF( datex + ": No Signal" );
                scont = True;
            }

            if( n >= 25 )  							// This allows me to begin stepping right before the issue
            {
                cs = 1;
            }
        }
    }
}

AddColumn( O, "Open", 1.7 );
AddColumn( H, "High", 1.7 );
AddColumn( L, "Low", 1.7 );
AddColumn( c, "Close", 1.7 );
AddColumn( buy, "Buy", 1.0 );
AddColumn( buyprice, "BuyPrice", 1.7 );
AddColumn( sell, "Sell", 1.0 );
AddColumn( SellPrice, "SellPrice", 1.7 );
AddColumn( by, "by", 1.0 );
AddColumn( Short, "Short", 1.0 );
AddColumn( ShortPrice, "ShortPrice", 1.7 );
AddColumn( Cover, "Cover", 1.0 );
AddColumn( CoverPrice, "CoverPrice", 1.7 );
AddColumn( sl, "sl", 1.0 );
AddColumn( rawbb, "Raw BB", 1.7 );
AddColumn( bb, "BB", 1.7 );
AddColumn( rawsb, "Raw SB", 1.7 );
AddColumn( sb, "SB", 1.7 );
AddColumn( bcnd1, "bc1", 1.0 );
AddColumn( bcnd2, "bc2", 1.0 );
AddColumn( bcnd3, "bc3", 1.0 );
AddColumn( bcnd4, "bc4", 1.0 );
AddColumn( bcnd5, "bc5", 1.0 );
AddColumn( bcnd6, "bc6", 1.0 );
AddColumn( bcnd7, "bc7", 1.0 );
AddColumn( scnd1, "sc1", 1.0 );
AddColumn( scnd2, "sc2", 1.0 );
AddColumn( scnd3, "sc3", 1.0 );
AddColumn( scnd4, "sc4", 1.0 );
AddColumn( scnd5, "sc5", 1.0 );
AddColumn( scnd6, "sc6", 1.0 );
AddColumn( scnd7, "sc7", 1.0 );

Any help you can provide will be greatly appreciated.

Thanks.

Matt

1 Like

Sorry, but I don't get where the "insult" is. Should you adjust your unfair judgement to something more co-operative, I may be able to help you further.

1 Like

@2mc It's unbelievable. It is your ninth day on this forum (so you are a rookie here) and you have already received a lot of help from Tomasz (the developer) and other users. Now when you have been instructed that your issue has been already discussed and explained many times, you feel insulted! I wish I could downvote replies and in general attitude like this.

2 Likes

I will accept your representation that you didn't insult me. However, it came across to me as insult. I felt it as an insult. That is how I felt when I read your words. However, I will chalk it off to the fact that English is not your first language.

Thank you for your kind offer to help. Thanks.

Also, I didn't report it earlier, but I also tried using "BackTestRegular" as well, and I get the same result. I know of nothing that I haven't done to debug this.

Thanks again for your help.

Matt

1 Like

Milosz,

Thank you for your comment. Yes, I have received a lot of help already. And, I have expressed loads of appreciation.

I follow all the rules. I do searching, reading, testing, tracing, stepping, etc.......all before I post. I spent days testing and analyzing before I post.

The sad fact is that the documentation is not clear. Or, perhaps not organized. You may be quite experienced with AmiBroker and you have gone through all of these issues. Good for you. Perhaps, you have forgotten how massive the documentation is and how it is spread all over the place.

And, this is NOT the 9th day I have been on this forum. It is the 9th day I registered. I have read this forum for a long time.

I guess you don't see how dismissive your own tone is. I don't come here to get insulted. I come here for help. I follow the rules and try to make my question as clear as possible without being too long. I'm not asking for someone to program for me. I'm just asking for a little kindness of pointing me in the right direction. Perhaps, kindness is too hard for you.

I do appreciate your rising to the defense of your friend. That's admirable. But, you have mischaracterized me. I don't appreciate that.

Matt

1 Like

Tomasz,

I do understand that Exploration is not Backtest. I was just trying to show all of what I did.

I failed to show the following from the Exploration:
image

In order for there to be a trade, the columns in highlighted in yellow must be true the day before entry is made (if price reaches entry price). In the yellow they are true. In the blue they are not all true.

This is the weirdness I was mentioning. I did not understand how the Backtester could institute a trade when all exploration, tracing, and stepping has shown there are no conditions for there to be a trade.

That is what I meant by weird difference between Explore and Backtest. I completely understand that they are different.

If you can help, I would tremendously appreciate it.

Thanks.

Matt

@2mc I'm sorry, I won't continue this discussion.

I simply couldn't understand how can you write that Tomasz insults you by trying to help you. You must understand, that Tomasz has been developing this software for over 20 years and simply can't repeat the same information to every new user, especially when it has been covered in the docs and on the forum, several times.

Although I haven't been very active on the forum recently, I have received almost 2 000 likes. Let's come back to this topic when you come close or beat me :wink:

2 Likes

Milosz,

It's not a competition. I readily admit you are accomplished and my superior.

I understand that Tomasz cannot repeat the same answer. I truly do. That is why I go to GREAT LENGTHS on my own, before I post.

One cannot question another's feelings. They are his. One may consider them irrational, but he cannot question that the other person feels the way they do. I felt insulted. I could go into detail as to how I felt that. But, I have already chalked it off to a language barrier.

I have gone through the docs. I have searched. Perhaps I'm not good at searching. Perhaps I am dense. Perhaps. Regardless, here I am having exhausted the tools THAT I KNOW OF to debug this issue on my own. None of the tools have revealed the answer. I'm certain I'm missing a tool or I have made an assumption that is wrong about how to use a particular tool. I don't know.

There are things I know. There are things I know that I don't know. And, then there are things that I don't know that I don't know. I'm just asking for some guidance.

And, I didn't ask Tomasz specifically to answer my question. I understand he is busy. I totally get it. I was hoping some other kind sole would help me.

Thanks in advance for anyone who can help.

Matt

@m2c, not sure this will help, but try adding this line before your main loop.

// Loop through all bars
Buy = Sell = Short = Cover = 0;

Otherwise, some of the Buy of Sell values may have some wrong values. I have seen it testing your formula simply putting this line at the end of code:

Filter = Buy;

I do not have your data, but testing on my machine, the exploration on a ticker over many bars showed some "random" values in the Buy column.

2 Likes

As perfect English native speaker you probably realize how condescending it is, don't you?

Sorry, I don't control your feelings. You are controlling them.

One thing I am sure is that there was absolutely nothing insulting in my reply. Totally opposite I was super helpful in ALL my replies (not only in this thread but in many others too). Yet again, I tried to help. But as old good saying goes "no good deed goes ever unpunished". Another lesson learned. I have better things to do on weekend.

2 Likes

Thanks, Beppe!!

That worked!! But, now I am perplexed as to why it worked.

What made you think of doing that? I'd like to understand where I missed that nugget of information. Or, I'd like to know where I missed asking myself this question.

Anyway, thank you so very, very much for the answer. I really appreciate it.

I'm glad to be back on my way. Your help was tremendous!!

Thanks again.

Matt

1 Like

Your code lacked the most important thing: INITIALIZATION. If you access array elements without initializing array first, you are going to get random data in all elements that you did not write the value to.

Mentioned here:

and here:

1 Like

Tomasz,

I apologize. I did not mean to be condescending. You yourself wrote the following to me:
"Also worth noting is that not everybody here (me included) are not native English speakers so sometimes we are not super skilled in reading nuances."

I took you at your word that you didn't insult me. Since I took it as an insult ...at first.. and since I had received your comment quoted above about not being a native English speaker and sometimes missing nuance, I then considered it to be the reason for my misunderstanding. I profusely apologize if that came across as condescending. I think when you see that totality of what I just described you will see that I truly wasn't being condescending. But, I cannot fault you, because that is how you felt. I can only profusely apologize for my miscommunication.

Thanks.

Matt

Tomasz,

Thank you very much. I appreciate it.

Here's where I went wrong:
I thought this handled the issue: SetOption( "EveryBarNullCheck", True );

Thanks for the clarification. I really appreciate it!

And great tip!!

Matt

I've simply seen many examples using custom loops, like in your code, and Buy, Sell, etc., must be initialized (as in the "Trailing Stop" snippet).

I checked the "Buy" filter because a long, long time ago, I learned the C programming language, and uninitialized variables were at the root of many weird problems. So I suspected this was the case here too!

2 Likes

Thanks, Beppe! This won't happen to me again, thanks to your help. And, also Tomasz had a slightly different - though equivalent result - solution. I appreciate both tips!

Thanks.

Matt

1 Like

I received notification that this post did not meet community standards. The instructions were:
"You can edit your post after 10 minutes, and it will be automatically unhidden."

I waited 10 minutes and then tried to edit my post, but I cannot see how to edit it. The pencil is gone.

How do I edit it?

5 posts were merged into an existing topic: Backtester and Exploration results mismatch