OSAKA sort -- replace with matrix

I am new to this site, and had appended this inquiry to a question on stock splits. Perhaps I should have separately posted this issue on OSAKA.

I use 6.28.0 professional real time 64 bit. I heavily use the 64 bit OSAKA routine for two dimensional sorts and extractions related to digital signal routines I have developed. Tomasz, thanks for updating it to 64 bit a while back. Its a great help for use by someone like me that is not a "natural" at programming. I have a Geortzel extraction routine that I wrote, and can share. It works, but uses OSAKA table for two dimensional sort and extractions. If converting to matrices would be much faster for these purposes, I would like to convert. I would like to convert anyway, just to learn more. If anyone has any code snippets or other that would help, please let me know. I can post my code, if anyone has time to provide any input....many thanks

1 Like

@entropy1969, quoting the AmiBroker members' zone main page:

Osaka plugin is open-source 3rd party plugin with 2D table support (now superseded by AmiBroker's native matrix support)

So, I suppose it will be faster!

You can search this forum for some example of usage of Matrixes, but probably the kind of samples you'll find will be quite different from your routine, but at least you can start to play with some matrix code.

Here some links to previous threads/posts:

In this last example, (not the most intuitive one...) I used a matrix to store some data and then sorted it multiple times.

In any case, if you'll post your existing code (if it isn't 1M lines...), maybe some of the most skilled users here could find the time to convert it and /or guide you in the conversion process.

Then you'll be able to tell us what version is faster.

2 Likes

Giuseppe:

Many thanks for the prompt and detailed reply with links. I will follow your suggestion, and post the simplest of my routines today or tomorrow. It is one of the shortest, not really too long. If I see how to modify for this routine, it should serve as a template for some of the other digital signal routines I have worked on. My existing code will also give something back to the forum community as it computes the Geortzel extraction, a digital routine Dennis Meyers PhD Math and trader in Chicago has discussed on his trading algorithm website, Meyers Analytics. He has many interesting ideas. He argues convincing ly (with papers for download on his site, I think free) that Geortzel algorithm can be a superior frequency discriminator to the Maximum Entropy Method in many cases. I use such discriminatory to tune my indicators to changing market conditions. Surprisingly, the Geortzel extraction is quite simple and few lines of code. MEM is much more involved.

Again, thank you and I will post code within today or tomorrow. Any help anyone can give is appreciated!!

Mike

Hi Entropy, where can i find the 64 bit OSAKA?

Login to the password protected members zone on the amibroker.com site. Immediately upon arriving there, a list of downloads are available. The Osaka 64 bit version is the third one down. Hope this helps....
Mike

1 Like

Hello everyone: [Thank you in advance]

I am contributing a loop with an OSAKA 64 bit sort for the Goertzel algorithm. The algorithm is a frequency detection routine that Chicago based trading advisor Dennis Meyers PhD (Math) asserts compares very favorably, in many cases, to Maximum Entropy method. You can find out more about this method on his site Meyers Analytics. He does, or at least used to have, free papers which compared the two on his site.
I am trying to get away from the Osaka 64 bit sort, and need help. I am not sure how to convert this code to matrix functions and variables. I use this Osaka routine for a long list of custom coded indicators.

First, my question.( I AM NOT A PROGRAMMER...., AND STRUGGLED THROUGH PUTTING THIS CODE TOGETHER, SO IT MAY NOT BE THE MOST EFFICIENT, BUT IT WORKS).

Then, a brief note about the code.

Then, the code.

  1. Can someone help me with some basic code for converting this loop, sort routine, and extraction of output to the new matrix functions?

I am donating the algorithm to the community. It is mathematically sound. Tomasz, if you believe this is worthwhile, you are free to add this, in whatever form with your programming expertise to Amibroker. It is a pretty sound frequency detection method.

  1. Brief bit about the code. For EACH price bar, it sorts through a range of values from 6 to 200 to detect the frequency with the highest amplitude (strongest frequency). The Osaka table is used to SORT AND EXTRACT the top 5 frequencies and associated amplitudes which can be put together to create an amplitude weighted indicator. (e.g., weighted avg, etc) Please note: if you change this range, the number in the "flattened price" section must be changed. It must always be double or more according to Meyers of the highest number in the range , so for 6 to 200, it must be 400 or more as shown

  2. Code snippet.

//This code checked as error free in AFL. It does not produce a final indicator. At the very end, you could //use these frequencies and their weightings to construct your own weighted average, or any other //weighted indicator

SetBarsRequired(sbrAll,0); 

SetChartOptions(0,chartShowArrows|chartShowDates);
_N(Title = StrFormat("{{NAME}} - {{INTERVAL}} {{DATE}} Open %g, Hi %g, Lo %g, Close %g (%.1f%%) {{VALUES}}", O, H, L, C, SelectedValue( ROC( C, 1 ) ) ));
//Plot( C, "Close", ParamColor("Color", colorBlack ), styleNoTitle | ParamStyle("Style") | GetPriceStyle() );

PI = 3.1415926;

Med = (H+L)/2;
k=0;
a = 0;
b = 0;
y = 0;
GA = 0;
AmpL = 0;

//earliest bar you want to take. the more bars, the longer it takes to process them
EarliestBar = 1;

//osaka plugin initialize

osInitialize();     
       
//osaka table create
tableID = osTabCreate();

//create osaka's table's columns
osTabAddColumn("amp", 1, tableID);
osTabAddColumn("GA", 1, tableID);
osTabAddColumn("bar", 1, tableID);

//gets "number_of_order"'s amplitude, use 1 for highest value, 2 for 2nd highest, etc.
//Use after sorting.
//pass MaxN-MinN+1 as freqCount
function GetAmplitude(bar_number, number_of_order, freqCount)
{
   return (osTabGet(freqCount*(bar_number - EarliestBar)+(number_of_order-1), 0, tableID));
}

//gets "number_of_order"'s frequency, use 1 for highest value, 2 for 2nd highest, etc.
//Use after sorting.
//pass MaxN-MinN+1 as freqCount
function GetFrequency(bar_number, number_of_order, freqCount)
{
   return (osTabGet(freqCount*(bar_number - EarliestBar)+(number_of_order-1), 1, tableID));
}

function flattenedprice(input)
{
	for(i = EarliestBar; i < BarCount; i++ )
	{
		if (i < 400 )    //In Meyers MESA vs. Goertzel paper, he indicates we should use at least twice the number of bars as MaxN  (400 = 2*200)
		{
			a = Med[i];   //need specified number of data points to begin calculation
			b = Null;
			y = Null;
		}
		else
		{
			g = 400;
			a = Med[i - g];
			b = (Med - a)/(g-1);
			y = Med - (a[i] + b[i]*(i-1));
		}
	}
	return y;
}

flt = flattenedprice(Med);
//Plot(flt,"MaxAmpL", colorBlue); // plots end flattened prices

// -- I declared these because we will need them more than once
MaxN = 200; 
MinN =  6;
// --
rown = 0; //row increment variable

for (n = MinN; n <= MaxN; n++)   //used minN and maxN instead of hardcoded values.
{
    for(i = EarliestBar; i <= BarCount - 1; i++ )
	{
		switch(i)
		{
			
			case 0 :
			{
				GA[0] = 0;
				break;
			}

			case 1 :
			{
				GA[1] = 2*cos(2*PI/n)*0 - 0 + flt[i];
				break;
			}

			default:
			{
				GA[i] = 2*cos(2*PI/n)*GA[i-1] - GA[i-2] + flt[i];
				// Amibroker uses radians. Goertzel formula uses 2 pi in this formulation
				//which equals 6.28318 radians.

				break;
			}

       }

		if(i == BarCount - 1)
		// According to Meyers article "MESA vs. Goertzel", Goertzel is
		// calculated after final item in flattened input (i.e., BarCount -1)
		{
          AmpL = (GA^2 + GA[i-1]^2 - 2*cos(2*PI/n)*GA* GA[i-1])/100;
          //we have to go through whole AmpL array and add values to Osaka table
          for (iteratorSet = EarliestBar; iteratorSet < BarCount; iteratorSet++)
          {
              //set values into table, 0 column - amplitude, 1 column - frequency, 2 column - barnumber
              osTabSetNumber(AmpL[iteratorSet], rown, 0, tableID);
              osTabSetNumber(n, rown, 1, tableID);
              osTabSetNumber(iteratorSet, rown, 2, tableID);
              rown++; //increment table row
          }
		}
		else
		{
			AmpL = Null;
		}
	}
	//Plot(AmpL ,"AmpL", colorBlue);
	 //if place Plot function here, I get sinusoids for ALL "n" values
}
//Plot(AmpL ,"AmpL", colorBlue);
// if place Plot function here, I get only final sinusoid for last "n"


//***  few tests and examples below ***//

//sorts table by barnumber (column 2) in ASCENDING ORDER (True)
// AND amplitude (column 0) in DESCENDING ORDER (False).
//see osaka's readme for details.

osTabSort(tableID, 2, True, 0, False);

for(i = EarliestBar; i <= BarCount - 1; i++ )
{
	CurrentFrequency[i] = GetFrequency(i, 1, MaxN-MinN+1);       //1st strongest
}

for(i = EarliestBar; i <= BarCount - 1; i++ )
{
	CurrentFrequencyA[i] = GetFrequency(i, 2, MaxN-MinN+1);      //2nd strongest
}

for(i = EarliestBar; i <= BarCount - 1; i++ )
{
	CurrentFrequencyB[i] = GetFrequency(i, 3, MaxN-MinN+1);     //3rd strongest
}

for(i = EarliestBar; i <= BarCount - 1; i++ )
{
	CurrentFrequencyC[i] = GetFrequency(i, 4, MaxN-MinN+1);    //4th strongest, and so on ...... 
}

for(i = EarliestBar; i <= BarCount - 1; i++ )
{
	CurrentFrequencyD[i] = GetFrequency(i, 5, MaxN-MinN+1);
}

DomCycle = Median(CurrentFrequency,3);
DomCycleA = Median(CurrentFrequencyA,3);
DomCycleB = Median(CurrentFrequencyB,3);
DomCycleC = Median(CurrentFrequencyC,3);
DomCycleD = Median(CurrentFrequencyD,3);

//************** These loops "extract" amplitudes for "weighting" should user desire to weight items by amplitude *****//


for(i = EarliestBar; i <= BarCount - 1; i++ )
{
	CurrentAmplitude[i] = GetAmplitude(i, 1, MaxN-MinN+1);
}

for(i = EarliestBar; i <= BarCount - 1; i++ )
{
	CurrentAmplitudeA[i] = GetAmplitude(i, 2, MaxN-MinN+1);
}

for(i = EarliestBar; i <= BarCount - 1; i++ )
{
	CurrentAmplitudeB[i] = GetAmplitude(i, 3, MaxN-MinN+1);
}

for(i = EarliestBar; i <= BarCount - 1; i++ )
{
	CurrentAmplitudeC[i] = GetAmplitude(i, 4, MaxN-MinN+1);
}

for(i = EarliestBar; i <= BarCount - 1; i++ )
{
	CurrentAmplitudeD[i] = GetAmplitude(i, 5, MaxN-MinN+1);
}


DomAmplitude = CurrentAmplitude;
DomAmplitudeA = CurrentAmplitudeA;
DomAmplitudeB = CurrentAmplitudeB;
DomAmplitudeC = CurrentAmplitudeC;
DomAmplitudeD = CurrentAmplitudeD;

SumAmplitude  = DomAmplitude + DomAmplitudeA + DomAmplitudeB + DomAmplitudeC + DomAmplitudeD; 

// use each respective amplitude divided by SumAmplitude times indicator output computed from respective associated cycle value should an amplitude weighted indicator be desired.*/

//always perform osTabDelete after you've done operations with table (usually at the very end of the AFL code)
osTabDelete(tableID);

@entropy1969 looking at your code I see that you use the Osaka sort to sort on multiple columns using a different order (ASC/DESC) for the two selected columns.
AFAIK, this is not possible using the default sort of Matrices that have only one ordering parameter (you can sort on multiple columns, but all are using the same direction).

For the rest, the code is simple to port to matrices, but the matrix sorting issue may require a different approach.

I tested your code with an average number of bars (2500/5000), and it seems to me that, moving from Osaka to matrices will not solve the slowness of the entire process that implements a series of nested loops over all the bars (where you fill the Osaka table). Populating a matrix will be probably faster but not enough.

Maybe that this the code should only be used with relatively few bars, otherwise, in my opinion, these nested loops seems a lot more important area to focus on, and I suspect that it will be one of the few cases where a DLL implementation will help.

Because it is not clear to me the ultimate goal of this formula (I mean if it is used to create an indicator, to backtest a trading system or only to export some data, etc.), my observations may be wrong.

Let's see if any other more experienced user can indicate a better way to simplify and speed up your code.

1 Like

@beppe - the way to sort multiple columns in different asc/desc order is just to multiply that column by -1, do the sort then multiply it back by -1.

2 Likes

@entropy1969

If looking at PDF e.g. from here then there is talk about price of 400. Another PDF here.

Yes, I know you doubled from MaxN = 200
But I don't think it is meant that way as you interpreted.


Then another quote from PDF (regarding formula -> flattening):

Upper formula is one (1) based. AFL is zero based.

So a (-> first data value) would be

a = x[0];
first = a;

And last would be

last = x[n-1];

As well as

b * (i-1)

in AFL would be just

 b * BarIndex()

(Barindex() array -> zero based)

So whole function (not a single looping code involved):

function flattenedprice( x ) {
	/// page 3 of
	/// @link http://meyersanalytics.com/publications2/AdaptNCycleGDFT.pdf
	/// @link http://meyersanalytics.com/publications2/MesaVsGDFT.pdf
	///
	/// @link https://forum.amibroker.com/t/osaka-sort-replace-with-matrix/6231/9
	n = BarCount;
	a = x[0];
	first = a;
	last = x[n-1];
	b = (last - first) / (n-1);
	y = x - (a + b*BarIndex());
	return y;
}

then rotate the rest of the time series such that the end point is now zero.

So applying upper function

Med = (H+L)/2;
flt = flattenedprice(Med);
Plot(flt,"flattenedprice", colorRed);

57

Also rest of your code does not look OK to me.


Anyway I only focus on conversion from Osaka to Matrix

First you initialize matrix before loop.

MaxN = 200;
MinN = 6;

rows = (MaxN-MinN+1)*BarCount;
cols = 3;
mat = Matrix(rows, cols);

within loop your replace osTabSetNumber by matrix element assignments

//osTabSetNumber( AmpL[iteratorSet], rown, 0, tableID );
//osTabSetNumber( n, rown, 1, tableID );
//osTabSetNumber( iteratorSet, rown, 2, tableID );
mat[rown][0] = AmpL[iteratorSet];
mat[rown][1] = n;
mat[rown][2] = iteratorSet;
rown++;

then the sorting (as suggested here) and using MxSortRows function.

/// OSAKA sorting commented
/// osTabSort(tableID, 2, True, 0, False);
///
/// instead sorting via MxSortRows
/// via suggestion by AB developer from here
/// @link https://forum.amibroker.com/t/osaka-sort-replace-with-matrix/6231/8
rownum = MxGetSize(mat, 0);
for( i = 0; i < rownum; i++ )
	mat[i][0] = -mat[i][0];
///
mat = MxSortRows(mat, True, 2, 0);
///
for( i = 0; i < rownum; i++ )
	mat[i][0] = -mat[i][0];

As for your amplitude and frequency functions (note: I added additional function argument!)

function GetAmplitude( mat, bar_number, number_of_order, freqCount )
{
    row =  freqCount * ( bar_number - EarliestBar ) + ( number_of_order - 1 );
    //return osTabGet(row, 0, tableID );
    return mat[ row ][ 0 ];
}
function GetFrequency( mat, bar_number, number_of_order, freqCount )
{
    row = freqCount * ( bar_number - EarliestBar ) + ( number_of_order - 1 );
    //return osTabGet(row, 1, tableID );
    return mat[ row ][ 1 ];
}

BTW this one of your code is pretty much overkill (n-times loop statements)


for(i = EarliestBar; i <= BarCount - 1; i++ )
{
	CurrentFrequency[i] = GetFrequency(i, 1, MaxN-MinN+1);       //1st strongest
}

for(i = EarliestBar; i <= BarCount - 1; i++ )
{
	CurrentFrequencyA[i] = GetFrequency(i, 2, MaxN-MinN+1);      //2nd strongest
}

for(i = EarliestBar; i <= BarCount - 1; i++ )
{
	CurrentFrequencyB[i] = GetFrequency(i, 3, MaxN-MinN+1);     //3rd strongest
}

for(i = EarliestBar; i <= BarCount - 1; i++ )
{
	CurrentFrequencyC[i] = GetFrequency(i, 4, MaxN-MinN+1);    //4th strongest, and so on ...... 
}

for(i = EarliestBar; i <= BarCount - 1; i++ )
{
	CurrentFrequencyD[i] = GetFrequency(i, 5, MaxN-MinN+1);
}

for(i = EarliestBar; i <= BarCount - 1; i++ )
{
	CurrentAmplitude[i] = GetAmplitude(i, 1, MaxN-MinN+1);
}

for(i = EarliestBar; i <= BarCount - 1; i++ )
{
	CurrentAmplitudeA[i] = GetAmplitude(i, 2, MaxN-MinN+1);
}

for(i = EarliestBar; i <= BarCount - 1; i++ )
{
	CurrentAmplitudeB[i] = GetAmplitude(i, 3, MaxN-MinN+1);
}

for(i = EarliestBar; i <= BarCount - 1; i++ )
{
	CurrentAmplitudeC[i] = GetAmplitude(i, 4, MaxN-MinN+1);
}

for(i = EarliestBar; i <= BarCount - 1; i++ )
{
	CurrentAmplitudeD[i] = GetAmplitude(i, 5, MaxN-MinN+1);
}

5 Likes

For what it is worth, I did not suggest inverting column using loops. You can negate single column in any matrix without any looping by multiplying matrix by transformation matrix that has 1's in all columns except the column you want to invert which has -1's.

m = MxFromString("[ 9, 1, 6; 40, 30, 20; 8, 7, 3; 3, 5, 1 ]");

printf("Input matrix : %s \n", MxToString( m ) ); 

col_to_negate = 1;

// create transformation matrix with 1's everywhere except column we want to negate
ng = MxSetBlock( m * 0 + 1, 0, MxGetSize( m, 0 ) - 1, col_to_negate, col_to_negate, -1 );

m = m * ng; // negate one column 

printf("Matrix with one column negated %s \n",  MxToString( m )  ); 

// do the sort MxSortRows

m = m * ng; // negate it back
4 Likes

Thanks for your suggestions.

I found a bit of time to do some tests on a modified version of the formula, using multiple GetPerformanceCounter() statements to measure (within the known limitations) the time to complete some parts of the code (I did not change any calculations, focusing only on the Osaka vs. matrix modifications.).

Using a matrix instead of an Osaka table is a lot faster in the main loop (where you calculate amplitude) to set the table/matrix values, in my tests, it took approx. 1/4 of time. Sorting time (including the extra negate step), on the other hand, seems comparable ( (by the way, the first negate operation can be done directly on each value during the matrix population loop).

Also, memory usage of matrices vs. Osaka is quite more efficient (take a look at the memory footprint used by the application during the execution).

So, even with the remaining slowness due to the nested loops used to calculate amplitudes, it seems to be a worthwhile improvement.

1 Like

Guiseppe:

Thank you for the prompt, pleasant and extensive reply. Sometimes on forums, some people's replies seem to be more about criticism and attitude than they do help. After all, if I knew, I would have already answered my question.

Thank you for a doing a benchmark comparison. I APPRECIATE YOUR EFFORT. I suspected it would be faster, but that is quite a bit. I am not a programmer, and just as I learned how to use the Osaka table from a code snippet or template someone sent me several years ago, would you mind sharing your code for the replacement of the Osaka table with the matrix functions?......then I could look at them side by side, and see how the matrix functions replace the table. I hope to use this to replace other Osaka table applications

Thank you very much.

Mike

Postscript:

In my "non - programmer, learn by doing" world, sometimes I wish someone would write an Amibroker AFL Programming for Dummies" book with very detailed code examples (especially for some of the more intricate AFL functions) AND EXPLANATIONS of how the code is structured in simple terminology. The explanations are important .These "Dummies" titled books are very common in the US, and popular for science and math. Amibroker has a large worldwide community, and this might be very useful.

2 Likes

That is because both Osaka plugin and AmiBroker use same (standard) algorithm - QuickSort.

1 Like

@entropy1969 thanks for your nice words.

The @Tomasz answer and the sample code provided by @fxshrat is essentially all you need to replace your Osaka calls with a matrix.

Here, we are very fortunate to have some very skilled programmers that share with us their knowledge and time!

I can not comment on the correctness of the implementation, since I have not examined the original document, but I suggest you carefully evaluate the additional suggestions of @fxshrat because he is usually right.

My code, that as I wrote, was only modified to get an overview of the times spent in different parts of the formula, is essentially a copy of what you posted, with just some extra lines.
For this reason, I think it may not be very useful to other people, so I will send it to you directly.

fxshrat:

I saw Guiseppe's response and thanked him, but in my hurry, had forgotten to say the same to you. I have, like most people, busy days, and am not beyond forgetting sometimes. I know you took quite a bit of time, and I appreciate it. At the moment I responded to Guiseppe, I was thinking about some replies I have seen on trading and other forums, and was referring in a global sense, not characterizing your response....that was the context I intended. We are lucky on this forum, and that's what I meant. I have always been very polite to Tomasz, Marcin, Guiseppe, and anyone else to my knowledge. Please do not assume that I am rude....I have never done that on here. If you believe I have been rude to you on some instance....just communicate with me, and I will try to clarify as I have in this case. That's all that is necessary.

3 Likes

Well, I thought this quote of yours

Sometimes on forums, some people's replies seem to be more about criticism and attitude than they do help.

was referring to other posts in this thread since you did not comment on other posts (which have taken time to create). So I thought you ignored on purpose just because your were given constructive help on code (without any personal criticisms). It was not clear to me that other forums were meant.

Then it was misunderstanding and I have deleted my previous post.

Again no one will make fun of beginners (because being beginner is not bad thing/sin to get punished for. Everyone was beginner some time in past). It is not personal thing to give comment on codes. As I said it is just about cold "boring" technical stuff.

4 Likes

fxshrt:

Thanks for deleting post and understanding. Appreciate input very much

Mike

fxshrt:

Have had more time to read and understand your points on the code. Will need a bit more time on the matrix part, but certainly understand your point on the overkill on those extractions from the table (i.e., the n times loops comment near the end of your first message.

Thank you for the help

Mike

[quote="entropy1969, post:19, topic:6231, full:true"]
fxshrt:

Have had more time to read and understand your points on the code. Will need a bit more time on the matrix part, but certainly understand your point on the overkill on those extractions from the table (i.e., the n times loops comment near the end of your first message.

Thank you for the help

Mike

Hello @beppe Sir,
can u share that code with me.
I am Also search rank with matrix.
Thanks in Advance.
Have a nice time.