Missing Monthly Bars

[jptrader]

Hello,

I would appreciate some help with the following issue I have found when backtesting on a monthly periodicity.

Some monthly bars are missing, such as that of the red arrow causing a 100% loss in CRUS. It happens with many other bars, as it is shown in the following screenshot.

However, after zooming in I have checked that CRUS date is fine on a daily basis.

Why is it happening? How could I restored the missing monthly bars?

Thank you for your help.

Regards.

[Tomasz]

It is hard to tell you what is wrong if you don't send the formula that you are using for the chart. But guessing from title line you have NULLS in Close and Low arrays (those -1e10 numbers). Nulls normally do NOT appear in OHLC arrays. You must be using that in your formula so you assign NULLS into certain array elements, something like this:

Plot( IIF( condition, Close, Null ), ....)

Such Nulls result in missing bar (Null used in Plot means "don't plot this bar").

The only way to get real help is to include the formula

[jptrader]

Tomasz,

Thank you for your promp answer.

This is the formula for the chart, it is the standard Price formula from the tab "Charts"

_SECTION_BEGIN("Price");
SetChartOptions(0,chartShowArrows|chartShowDates);
_N(Title = StrFormat("{{NAME}} - {{INTERVAL}} {{DATE}} Open %g, Hi %g, Lo %g, Close %g (%.1f%%) {{VALUES}}", O, H, L, C, SelectedValue( ROC( C, 1 ) ) ));
Plot( C, "Close", ParamColor("Color", colorDefault ), styleNoTitle | ParamStyle("Style") | GetPriceStyle() ); 
_SECTION_END();

I have realized that I also have weird events in Volume for the same bars that monthly prices are missing.

Again, I am using the standard Volume formula

_SECTION_BEGIN("Volume");
Plot( Volume, _DEFAULT_NAME(), ParamColor("Color", colorBlueGrey ), ParamStyle( "Style", styleHistogram | styleOwnScale | styleThick, maskHistogram  ) );
_SECTION_END();

I have just updated the database from Norgate pluggin and everything has gone to normal.

Until I run the following code

// Parameters

Positions = Param("Positions", 25, 1, 50, 1);


// --- Backtester settings --- 

SetOption("InitialEquity", 10000);
SetOption("MaxOpenshort", 0);
SetOption("MaxOpenLong", Positions);
SetOption("MaxOpenPositions", Positions);
SetOption("AllowPositionShrinking", True);
SetOption("MinShares", 0);
SetOption("HoldMinBars", 1);
SetOption("CommissionMode", 1);
SetOption("CommissionAmount", 0.2);


// --- Constants ---


// --- Variables ---

MktMomPer = 6;
Rank_Limit = Positions; 


// --- Formulas ---


// --- Ranking ---

#include_once "Formulas\Norgate Data\Norgate Data Functions.afl"
WlNum = GetOption("FilterIncludeWatchlist"); 
List  = CategoryGetSymbols(categoryWatchlist, WlNum); 

if ( Status("stocknum") == 0 ) 
{ 
    StaticVarRemove("Value*");

    for (n = 0; (Symbol = StrExtract(List, n))  != "";  n++) 
    { 
        SetForeign (Symbol);
        Ticker         = WriteIf(StrLen(Name()) < 8, Name(), Strleft(Name(), StrLen(Name()) - 7));
        FundValue  = Ref(Foreign(Ticker + "_ART_ROE", "V", 2), -1);
        FundScore  = FundValue + ROC(Close, 1)/1000;
        Condition    = NorgateIndexConstituentTimeSeries("$SPX") AND FundValue > 0; 
        Score          = IIf(Condition, FundScore, 0); 
        RestorePriceArrays(); 
        StaticVarSet ("Value" + Symbol, FundValue);  
        StaticVarSet ("Score" + Symbol, Score);  
    } 
    StaticVarGenerateRanks("Rank", "Score", 0, 1224 );
} 

Symbol = Name(); 
Value = StaticVarGet("Value" + Symbol);      
Score = StaticVarGet("Score" + Symbol);
Rank = StaticVarGet("RankScore" + Symbol);


// --- Setup ---

Setup = 1; // ROC(Foreign("$SPX", "C"), MktMomPer) > 0;


// --- Conditions ---

BuyCondition1 = Setup AND Rank <= Rank_Limit;
SellCondition1 = Rank > Rank_Limit;


// --- Trading ---

Buy = BuyCondition1;
Sell = SellCondition1;

BuyPrice  = Open;
SellPrice  = Open; 

SetPositionSize(100/Positions, spsPercentOfEquity);
SetTradeDelays(1, 1, 1, 1);


// --- Exploration  ---

Filter = 1;
AddColumn(Ref(NorgateIndexConstituentTimeSeries("$SPX"), -1), "Index", 1.0); 
AddColumn(Ref(Value, -1), "Value", 1.3); 
AddColumn(Ref(Score, -1), "Score", 1.3); 
AddColumn(Ref(Rank, -1), "Rank", 1.0);

And everything goes crazy again...

What do you think?

[Tomasz]

The formula by itself does not have ability to change quotations (is not able to write anything to the database).
As I wrote before, in the title bar I can see you have -1e10 in LOW and CLOSE.
This has to come from somewhere. If you are using Norgate plugin, then plugin has total control over database. And I would start looking at the data.

You might go to Symbol->Quote Editor and look at the raw data if there is something wrong with the data. Also you might see if you have any filtering / padding enabled in "View" menu.

[jptrader]

Indeed, the database does contains -1e10 prices in the affected months. The chart represents the data correctly. The problem is that somehow the data is mutating when I backtest. I will try to ilustrate the process with screenshots, using symbol MSFT and AJG as examples from two different backtests.

1. Updating the database and checking that MSFT and AJG data look correct in 2000.
   

   
   
   
   

2. Running the backtest and finding Long (ruin) trades
   

   
   
   
   

3. Finding that the data from the chart have changed without other action from the user than backtesting. Some monthly bars are missing (-1e10 value) and volume has weird spikes.
   

   
   
   
   

4. Verifying that the raw data from Symbol > Quote Editor have one day with weird values that were not present in the original data.
   

   
   
   
   

I do have pad-and-align option enabled because the code uses Foreign() functions.

Anyway, I have tried it after disabling it and I get the same problem with the changing data.

What should I do?

[Tomasz]

-1e10 values (Nulls) MUST NOT be present in the raw OHLC data. Something is wrong with the data.

Please write to Norgate directly (since you seem to use them as data vendor).

[NorgateData]

We've never put out NULLs so I think there's some sort of corruption going on with your database somewhere. I'll send an email so we can diagnose this further.

[Tomasz]

You might create a NEW DATABASE and see if issue persists.

[jptrader]

Finally, the Norgate team assisted me by email, and we resolved the issue. The database had become corrupted on my computer, and the solution was to create a new one. Now it works as intended.

Thank you for your help.