J shaped patterns. Machine Learning

Easy to see with the naked eye, not so easy to describe in English, let alone AFL. I think I should start with a 'best fit' moving average. How would I do that please?

x

4 Likes

For the curve:

image
You can modify this AFL from the Library by using the section of the code that identifes the Cup (remove the Handle section).

Then set criteria to code the rest as per your feel in order to get the desired J-shape. Reverse all the criterias to get an inverted J-shape (if required).

image

4 Likes

A better way would be training a ML model to recognize this (any) pattern.
I trained a simple logistic regression model and used the learned parameters to detect this pattern ( using only close prices ) .

Here is the result of an exploration against multiple symbols
image

Here is the detection part of the code (might need some tweaking for faster execution )

// J-Pattern Detection
// Aron Pipa April 2019
// Version 1.00

function cat( x, y )
{
    // concatenate horizontally
    rows = MxGetSize( x, 0 ) ;

    xcols = MxGetSize( x, 1 ) ;
    ycols = MxGetSize( y, 1 ) ;
    cols = xcols + ycols ;
    z = Matrix( rows , cols ) ;

    for( i = 0 ; i < rows; i ++ )
    {
        for( j = 0 ; j < cols; j ++ )
        {

            if( j >= xcols )
                value = y[i][j - xcols];
            else
                value = x[i][j];

            z[i][j] = value;
        }
    }

    return z ;
}
function predict( x, w, b )
{
    n = MxGetSize( x, 1 ) ;
    b = b@Matrix( 1, n , 1 ) ;
    z = w@x + b;
    return 1 / ( 1 + exp( 0 - z ) ) ;
}

// import target pattern close prices;
target = "{{4.18,4.10,4.05,3.96,4.05,4.09,3.97,4.01,4,4.03,4.09,4,4.08,4.10,4.08,4.14,4.24,4.41,4.38,4.39,4.59,4.70,4.92,5.41}}" ;
target = MxFromString( target ) ;
pattern = MxTranspose( target ) ;
pattern_length = MxGetSize( pattern , 0 );

//scale target pattern to [0 1] range
sorted_pattern = MxSortRows( pattern ) ;
m = pattern_length - 1;
minp = sorted_pattern[0][0];
maxp = sorted_pattern[m][0];
pattern = ( pattern - minp ) / ( maxp - minp ) ;

// visualize target pattern
XYChartSetAxis( chart = "prediction", "", "[red] mean of matching patterns   [yellow] actual pattern  ", styleLine | styleDots );
if( Status( "stocknum" ) == 0 )
{
    for( i = 0 ; i < pattern_length ; i ++ ){
        XYChartAddPoint( chart, "",  i, pattern [i][0], colorYellow, colorYellow );
    }
}


// use parameters learned form trained model
w = "{{ -0.9128377, -1.450928, -1.190372, -1.587574, -1.37189, -1.309276, -2.290018, -1.739202, -1.684659, -1.494346, -1.563871, -1.958653, -1.705168, -1.259665, -1.238384, -0.9963449, -0.4883861, -0.03951919, -0.02338271, -0.1006666, 0.4559279, 0.8036441, 1.237101, 3.116307 }}";
b = "{{ 2.153458 }}";
w = MxFromString( w ) ;
b = MxFromString( b ) ;


// test
SetBarsRequired( sbrAll ) ;
Buy = Sell = Short = Cover = 0 ;
PositionSize = 1000; 
SetBacktestMode ( backtestRegularRawMulti ) ; 
ApplyStop( stopTypeNBar, stopModeBars, 5, True );


threshold = Optimize( "matching probability" , 0.99, 0.59, 0.99, 0.1 ) ;   // probability that a pattern form test dataset matches the target pattern

x = xx = Matrix( pattern_length, 1 ) ;
n = 0 ;  // number of positively classified patterns

// extract patterns form historical data
// scale to [0 1] range
// classify extracted pattern
// concatenate all matching patterns into one matrix 
for( j = pattern_length ; j < BarCount ; j ++ )
{

    k = 0 ;
    xmin = xmax = Close[j] ;
    for( i = j - pattern_length + 1 ; i <= j ; i ++ )
    {
        x[k][0] = Close [i];
        xmin = Min( xmin , Close [i] ) ;
        xmax = Max( xmax , Close [i] ) ;
        k++;
    }
    x = ( x - xmin ) / ( xmax - xmin ) ;

    y = predict( x, w, b ) > threshold ; // classify current pattern

    buy [j] = y[0][0];

    if( y[0][0] ) 
    {
        n++;
        xx = cat( xx , x ) ;  
    }

}

SetChartOptions( 0,  chartShowDates | chartShowArrows ) ;
Plot( C, "", colordefault , styleBar );

Filter = Status( "lastbarinrange" ) ;
AddColumn( n , "matching patterns") ;
if( n )
{
	// calculate the average of matching patterns 
    xx = MxGetBlock( xx , 0, pattern_length - 1 ,1 , n ) ;
    detected = MxTranspose( xx ) ;
    mean = 1 / n * ( Matrix( 1, n, 1 ) @ detected )  ;
    
    // visualize average matching pattern
    XYChartAddPoint( chart, "",  Null, Null ) ;
    for( i = 0 ; i < pattern_length ; i ++ ){
		XYChartAddPoint( chart, "",  i, mean [0][i], colorRed, colorRed );
    }
}



39 Likes

@aron thank you for sharing this technique!

1 Like

@aron great stuff :+1: - thanks for sharing

6 Likes

Thank you for this very interesting post.
Could you please elaborate further on how you train your logistic regression, in particular what features are you using?

1 Like

Training is fairly simple,
You create a training dataset using the same number of copies of the scaled target pattern and random patterns, creating 2 classes, the exact pattern and the random pattern.

2 Likes

So if I properly understand:

  • you create n copies of the pattern that you label "pattern"
  • you generate n random sequences, using random walk as an example, that you label "non pattern"
  • you train a classification algo
    I also guess that you are doing this process outside of AB and then import the weights.

Exactly, n copies of target pattern and n random sequences so that lables "pattern" and "non pattern" are balanced.
I have done this in AFL so that I can make use of AB exploration and backtest functionalities

1 Like

Nice work!!! Thank you.

It is worth mentioning that @aron first wrote about his Machine Learning system and attached sample AFL and CSV files here: Machine Learning: my first system (Old Yahoo Group)

4 Likes

Hello @aron

Great stuff, I am trying to study and understand what you are doing.

May I ask, how do you find the w values? Is it from another piece of AFL code?

@aron, @milosz and @Tomasz, what would be some good readings to get the necessary background to understand what @aron does (Machine Learning for Amibroker)?

I did look at the yahoo link @milosz shared. It uses string manipulation a lot and I think this is replaced with matrixes in this thread's code. So maybe the yahoo thread is not the ideal place to start.

The training part is not included , I think should post some updated code in this forum, since referencing old Yahoo group is not a good option for those interested.

I am merely a practitioner in this field - not even good enough to give advice but I have used the following resources:
https://www.coursera.org/learn/machine-learning?
https://www.deeplearning.ai/
http://www.deeplearningbook.org/
https://karpathy.github.io/
https://machinelearningmastery.com/

12 Likes

Oh yes! I think a small example that shows the steps would be great.

1 Like

The posts showing new, different or alternative ways of utilizing AmiBroker's capabilities are the most interesting and inspiring for me (and probably for many other users too). Machine Learning is definitely among the hottest and most promising topics nowadays.

@aron - I'm sure that if you were so kind to present a basic tutorial showing how do you manage the whole process in AFL (including the training part) many users (among them me) would be really grateful for that. Thanks :slight_smile:

7 Likes

Hello @aron

I am studying your code. May I ask, this code will examine patterns that have a length of exactly 24 bars. So if there is a J shape that spans over 10 bars or 120 bars, it will not detect it. Correct?

The scaling applies to prices (y axis) only. It does not apply to the number of bars.

Correct. The patterns examined are all of the same length (bars) as the target pattern. The length of the pattern determines the length of the weights learned in training, and you cannot use these weights to examine a pattern of different length.

1 Like

May I ask, have you developed other techniques, that will detect patterns of different length?

You can create target patterns of different length or shape and use a loop to train and detect each

Hello @aron

I came to this point and I am strugling to understand what it does (why it does it):

function predict( x, w, b )
{
    n = MxGetSize( x, 1 ) ;
    b = b@Matrix( 1, n , 1 ) ;
    z = w@x + b;
    return 1 / ( 1 + exp( 0 - z ) ) ;
}

Actually the problem is these two lines:

    z = w@x + b;
    return 1 / ( 1 + exp( 0 - z ) ) ;

x is the matrix holding price values from the chart piece to be examined. w and b hold the "learned" parameters. After that I am lost.

Is is easy to explain in simple terms what is happening here?

I am about half way through the coursera course you recommended. Are these explained there?