Hi Entropy, where can i find the 64 bit OSAKA?
Login to the password protected members zone on the amibroker.com site. Immediately upon arriving there, a list of downloads are available. The Osaka 64 bit version is the third one down. Hope this helps....
Mike
1 Like
Hello everyone: [Thank you in advance]
I am contributing a loop with an OSAKA 64 bit sort for the Goertzel algorithm. The algorithm is a frequency detection routine that Chicago based trading advisor Dennis Meyers PhD (Math) asserts compares very favorably, in many cases, to Maximum Entropy method. You can find out more about this method on his site Meyers Analytics. He does, or at least used to have, free papers which compared the two on his site.
I am trying to get away from the Osaka 64 bit sort, and need help. I am not sure how to convert this code to matrix functions and variables. I use this Osaka routine for a long list of custom coded indicators.
First, my question.( I AM NOT A PROGRAMMER...., AND STRUGGLED THROUGH PUTTING THIS CODE TOGETHER, SO IT MAY NOT BE THE MOST EFFICIENT, BUT IT WORKS).
Then, a brief note about the code.
Then, the code.
- Can someone help me with some basic code for converting this loop, sort routine, and extraction of output to the new matrix functions?
I am donating the algorithm to the community. It is mathematically sound. Tomasz, if you believe this is worthwhile, you are free to add this, in whatever form with your programming expertise to Amibroker. It is a pretty sound frequency detection method.
-
Brief bit about the code. For EACH price bar, it sorts through a range of values from 6 to 200 to detect the frequency with the highest amplitude (strongest frequency). The Osaka table is used to SORT AND EXTRACT the top 5 frequencies and associated amplitudes which can be put together to create an amplitude weighted indicator. (e.g., weighted avg, etc) Please note: if you change this range, the number in the "flattened price" section must be changed. It must always be double or more according to Meyers of the highest number in the range , so for 6 to 200, it must be 400 or more as shown
-
Code snippet.
//This code checked as error free in AFL. It does not produce a final indicator. At the very end, you could //use these frequencies and their weightings to construct your own weighted average, or any other //weighted indicator
SetBarsRequired(sbrAll,0);
SetChartOptions(0,chartShowArrows|chartShowDates);
_N(Title = StrFormat("{{NAME}} - {{INTERVAL}} {{DATE}} Open %g, Hi %g, Lo %g, Close %g (%.1f%%) {{VALUES}}", O, H, L, C, SelectedValue( ROC( C, 1 ) ) ));
//Plot( C, "Close", ParamColor("Color", colorBlack ), styleNoTitle | ParamStyle("Style") | GetPriceStyle() );
PI = 3.1415926;
Med = (H+L)/2;
k=0;
a = 0;
b = 0;
y = 0;
GA = 0;
AmpL = 0;
//earliest bar you want to take. the more bars, the longer it takes to process them
EarliestBar = 1;
//osaka plugin initialize
osInitialize();
//osaka table create
tableID = osTabCreate();
//create osaka's table's columns
osTabAddColumn("amp", 1, tableID);
osTabAddColumn("GA", 1, tableID);
osTabAddColumn("bar", 1, tableID);
//gets "number_of_order"'s amplitude, use 1 for highest value, 2 for 2nd highest, etc.
//Use after sorting.
//pass MaxN-MinN+1 as freqCount
function GetAmplitude(bar_number, number_of_order, freqCount)
{
return (osTabGet(freqCount*(bar_number - EarliestBar)+(number_of_order-1), 0, tableID));
}
//gets "number_of_order"'s frequency, use 1 for highest value, 2 for 2nd highest, etc.
//Use after sorting.
//pass MaxN-MinN+1 as freqCount
function GetFrequency(bar_number, number_of_order, freqCount)
{
return (osTabGet(freqCount*(bar_number - EarliestBar)+(number_of_order-1), 1, tableID));
}
function flattenedprice(input)
{
for(i = EarliestBar; i < BarCount; i++ )
{
if (i < 400 ) //In Meyers MESA vs. Goertzel paper, he indicates we should use at least twice the number of bars as MaxN (400 = 2*200)
{
a = Med[i]; //need specified number of data points to begin calculation
b = Null;
y = Null;
}
else
{
g = 400;
a = Med[i - g];
b = (Med - a)/(g-1);
y = Med - (a[i] + b[i]*(i-1));
}
}
return y;
}
flt = flattenedprice(Med);
//Plot(flt,"MaxAmpL", colorBlue); // plots end flattened prices
// -- I declared these because we will need them more than once
MaxN = 200;
MinN = 6;
// --
rown = 0; //row increment variable
for (n = MinN; n <= MaxN; n++) //used minN and maxN instead of hardcoded values.
{
for(i = EarliestBar; i <= BarCount - 1; i++ )
{
switch(i)
{
case 0 :
{
GA[0] = 0;
break;
}
case 1 :
{
GA[1] = 2*cos(2*PI/n)*0 - 0 + flt[i];
break;
}
default:
{
GA[i] = 2*cos(2*PI/n)*GA[i-1] - GA[i-2] + flt[i];
// Amibroker uses radians. Goertzel formula uses 2 pi in this formulation
//which equals 6.28318 radians.
break;
}
}
if(i == BarCount - 1)
// According to Meyers article "MESA vs. Goertzel", Goertzel is
// calculated after final item in flattened input (i.e., BarCount -1)
{
AmpL = (GA^2 + GA[i-1]^2 - 2*cos(2*PI/n)*GA* GA[i-1])/100;
//we have to go through whole AmpL array and add values to Osaka table
for (iteratorSet = EarliestBar; iteratorSet < BarCount; iteratorSet++)
{
//set values into table, 0 column - amplitude, 1 column - frequency, 2 column - barnumber
osTabSetNumber(AmpL[iteratorSet], rown, 0, tableID);
osTabSetNumber(n, rown, 1, tableID);
osTabSetNumber(iteratorSet, rown, 2, tableID);
rown++; //increment table row
}
}
else
{
AmpL = Null;
}
}
//Plot(AmpL ,"AmpL", colorBlue);
//if place Plot function here, I get sinusoids for ALL "n" values
}
//Plot(AmpL ,"AmpL", colorBlue);
// if place Plot function here, I get only final sinusoid for last "n"
//*** few tests and examples below ***//
//sorts table by barnumber (column 2) in ASCENDING ORDER (True)
// AND amplitude (column 0) in DESCENDING ORDER (False).
//see osaka's readme for details.
osTabSort(tableID, 2, True, 0, False);
for(i = EarliestBar; i <= BarCount - 1; i++ )
{
CurrentFrequency[i] = GetFrequency(i, 1, MaxN-MinN+1); //1st strongest
}
for(i = EarliestBar; i <= BarCount - 1; i++ )
{
CurrentFrequencyA[i] = GetFrequency(i, 2, MaxN-MinN+1); //2nd strongest
}
for(i = EarliestBar; i <= BarCount - 1; i++ )
{
CurrentFrequencyB[i] = GetFrequency(i, 3, MaxN-MinN+1); //3rd strongest
}
for(i = EarliestBar; i <= BarCount - 1; i++ )
{
CurrentFrequencyC[i] = GetFrequency(i, 4, MaxN-MinN+1); //4th strongest, and so on ......
}
for(i = EarliestBar; i <= BarCount - 1; i++ )
{
CurrentFrequencyD[i] = GetFrequency(i, 5, MaxN-MinN+1);
}
DomCycle = Median(CurrentFrequency,3);
DomCycleA = Median(CurrentFrequencyA,3);
DomCycleB = Median(CurrentFrequencyB,3);
DomCycleC = Median(CurrentFrequencyC,3);
DomCycleD = Median(CurrentFrequencyD,3);
//************** These loops "extract" amplitudes for "weighting" should user desire to weight items by amplitude *****//
for(i = EarliestBar; i <= BarCount - 1; i++ )
{
CurrentAmplitude[i] = GetAmplitude(i, 1, MaxN-MinN+1);
}
for(i = EarliestBar; i <= BarCount - 1; i++ )
{
CurrentAmplitudeA[i] = GetAmplitude(i, 2, MaxN-MinN+1);
}
for(i = EarliestBar; i <= BarCount - 1; i++ )
{
CurrentAmplitudeB[i] = GetAmplitude(i, 3, MaxN-MinN+1);
}
for(i = EarliestBar; i <= BarCount - 1; i++ )
{
CurrentAmplitudeC[i] = GetAmplitude(i, 4, MaxN-MinN+1);
}
for(i = EarliestBar; i <= BarCount - 1; i++ )
{
CurrentAmplitudeD[i] = GetAmplitude(i, 5, MaxN-MinN+1);
}
DomAmplitude = CurrentAmplitude;
DomAmplitudeA = CurrentAmplitudeA;
DomAmplitudeB = CurrentAmplitudeB;
DomAmplitudeC = CurrentAmplitudeC;
DomAmplitudeD = CurrentAmplitudeD;
SumAmplitude = DomAmplitude + DomAmplitudeA + DomAmplitudeB + DomAmplitudeC + DomAmplitudeD;
// use each respective amplitude divided by SumAmplitude times indicator output computed from respective associated cycle value should an amplitude weighted indicator be desired.*/
//always perform osTabDelete after you've done operations with table (usually at the very end of the AFL code)
osTabDelete(tableID);
@entropy1969 looking at your code I see that you use the Osaka sort to sort on multiple columns using a different order (ASC/DESC) for the two selected columns.
AFAIK, this is not possible using the default sort of Matrices that have only one ordering parameter (you can sort on multiple columns, but all are using the same direction).
For the rest, the code is simple to port to matrices, but the matrix sorting issue may require a different approach.
I tested your code with an average number of bars (2500/5000), and it seems to me that, moving from Osaka to matrices will not solve the slowness of the entire process that implements a series of nested loops over all the bars (where you fill the Osaka table). Populating a matrix will be probably faster but not enough.
Maybe that this the code should only be used with relatively few bars, otherwise, in my opinion, these nested loops seems a lot more important area to focus on, and I suspect that it will be one of the few cases where a DLL implementation will help.
Because it is not clear to me the ultimate goal of this formula (I mean if it is used to create an indicator, to backtest a trading system or only to export some data, etc.), my observations may be wrong.
Let's see if any other more experienced user can indicate a better way to simplify and speed up your code.
1 Like
@beppe - the way to sort multiple columns in different asc/desc order is just to multiply that column by -1, do the sort then multiply it back by -1.
2 Likes
If looking at PDF e.g. from here then there is talk about price of 400. Another PDF here.
Yes, I know you doubled from MaxN = 200
But I don't think it is meant that way as you interpreted.
Then another quote from PDF (regarding formula -> flattening):
Upper formula is one (1) based. AFL is zero based.
So a (-> first data value) would be
a = x[0];
first = a;
And last would be
last = x[n-1];
As well as
b * (i-1)
in AFL would be just
b * BarIndex()
(Barindex() array -> zero based)
So whole function (not a single looping code involved):
function flattenedprice( x ) {
/// page 3 of
/// @link http://meyersanalytics.com/publications2/AdaptNCycleGDFT.pdf
/// @link http://meyersanalytics.com/publications2/MesaVsGDFT.pdf
///
/// @link https://forum.amibroker.com/t/osaka-sort-replace-with-matrix/6231/9
n = BarCount;
a = x[0];
first = a;
last = x[n-1];
b = (last - first) / (n-1);
y = x - (a + b*BarIndex());
return y;
}
then rotate the rest of the time series such that the end point is now zero.
So applying upper function
Med = (H+L)/2;
flt = flattenedprice(Med);
Plot(flt,"flattenedprice", colorRed);
Also rest of your code does not look OK to me.
Anyway I only focus on conversion from Osaka to Matrix
First you initialize matrix before loop.
MaxN = 200;
MinN = 6;
rows = (MaxN-MinN+1)*BarCount;
cols = 3;
mat = Matrix(rows, cols);
within loop your replace osTabSetNumber by matrix element assignments
//osTabSetNumber( AmpL[iteratorSet], rown, 0, tableID );
//osTabSetNumber( n, rown, 1, tableID );
//osTabSetNumber( iteratorSet, rown, 2, tableID );
mat[rown][0] = AmpL[iteratorSet];
mat[rown][1] = n;
mat[rown][2] = iteratorSet;
rown++;
then the sorting (as suggested here) and using MxSortRows function.
/// OSAKA sorting commented
/// osTabSort(tableID, 2, True, 0, False);
///
/// instead sorting via MxSortRows
/// via suggestion by AB developer from here
/// @link https://forum.amibroker.com/t/osaka-sort-replace-with-matrix/6231/8
rownum = MxGetSize(mat, 0);
for( i = 0; i < rownum; i++ )
mat[i][0] = -mat[i][0];
///
mat = MxSortRows(mat, True, 2, 0);
///
for( i = 0; i < rownum; i++ )
mat[i][0] = -mat[i][0];
As for your amplitude and frequency functions (note: I added additional function argument!)
function GetAmplitude( mat, bar_number, number_of_order, freqCount )
{
row = freqCount * ( bar_number - EarliestBar ) + ( number_of_order - 1 );
//return osTabGet(row, 0, tableID );
return mat[ row ][ 0 ];
}
function GetFrequency( mat, bar_number, number_of_order, freqCount )
{
row = freqCount * ( bar_number - EarliestBar ) + ( number_of_order - 1 );
//return osTabGet(row, 1, tableID );
return mat[ row ][ 1 ];
}
BTW this one of your code is pretty much overkill (n-times loop statements)
for(i = EarliestBar; i <= BarCount - 1; i++ )
{
CurrentFrequency[i] = GetFrequency(i, 1, MaxN-MinN+1); //1st strongest
}
for(i = EarliestBar; i <= BarCount - 1; i++ )
{
CurrentFrequencyA[i] = GetFrequency(i, 2, MaxN-MinN+1); //2nd strongest
}
for(i = EarliestBar; i <= BarCount - 1; i++ )
{
CurrentFrequencyB[i] = GetFrequency(i, 3, MaxN-MinN+1); //3rd strongest
}
for(i = EarliestBar; i <= BarCount - 1; i++ )
{
CurrentFrequencyC[i] = GetFrequency(i, 4, MaxN-MinN+1); //4th strongest, and so on ......
}
for(i = EarliestBar; i <= BarCount - 1; i++ )
{
CurrentFrequencyD[i] = GetFrequency(i, 5, MaxN-MinN+1);
}
for(i = EarliestBar; i <= BarCount - 1; i++ )
{
CurrentAmplitude[i] = GetAmplitude(i, 1, MaxN-MinN+1);
}
for(i = EarliestBar; i <= BarCount - 1; i++ )
{
CurrentAmplitudeA[i] = GetAmplitude(i, 2, MaxN-MinN+1);
}
for(i = EarliestBar; i <= BarCount - 1; i++ )
{
CurrentAmplitudeB[i] = GetAmplitude(i, 3, MaxN-MinN+1);
}
for(i = EarliestBar; i <= BarCount - 1; i++ )
{
CurrentAmplitudeC[i] = GetAmplitude(i, 4, MaxN-MinN+1);
}
for(i = EarliestBar; i <= BarCount - 1; i++ )
{
CurrentAmplitudeD[i] = GetAmplitude(i, 5, MaxN-MinN+1);
}
5 Likes
For what it is worth, I did not suggest inverting column using loops. You can negate single column in any matrix without any looping by multiplying matrix by transformation matrix that has 1's in all columns except the column you want to invert which has -1's.
m = MxFromString("[ 9, 1, 6; 40, 30, 20; 8, 7, 3; 3, 5, 1 ]");
printf("Input matrix : %s \n", MxToString( m ) );
col_to_negate = 1;
// create transformation matrix with 1's everywhere except column we want to negate
ng = MxSetBlock( m * 0 + 1, 0, MxGetSize( m, 0 ) - 1, col_to_negate, col_to_negate, -1 );
m = m * ng; // negate one column
printf("Matrix with one column negated %s \n", MxToString( m ) );
// do the sort MxSortRows
m = m * ng; // negate it back
4 Likes
Thanks for your suggestions.
I found a bit of time to do some tests on a modified version of the formula, using multiple GetPerformanceCounter() statements to measure (within the known limitations) the time to complete some parts of the code (I did not change any calculations, focusing only on the Osaka vs. matrix modifications.).
Using a matrix instead of an Osaka table is a lot faster in the main loop (where you calculate amplitude) to set the table/matrix values, in my tests, it took approx. 1/4 of time. Sorting time (including the extra negate step), on the other hand, seems comparable ( (by the way, the first negate operation can be done directly on each value during the matrix population loop).
Also, memory usage of matrices vs. Osaka is quite more efficient (take a look at the memory footprint used by the application during the execution).
So, even with the remaining slowness due to the nested loops used to calculate amplitudes, it seems to be a worthwhile improvement.
1 Like
Guiseppe:
Thank you for the prompt, pleasant and extensive reply. Sometimes on forums, some people's replies seem to be more about criticism and attitude than they do help. After all, if I knew, I would have already answered my question.
Thank you for a doing a benchmark comparison. I APPRECIATE YOUR EFFORT. I suspected it would be faster, but that is quite a bit. I am not a programmer, and just as I learned how to use the Osaka table from a code snippet or template someone sent me several years ago, would you mind sharing your code for the replacement of the Osaka table with the matrix functions?......then I could look at them side by side, and see how the matrix functions replace the table. I hope to use this to replace other Osaka table applications
Thank you very much.
Mike
Postscript:
In my "non - programmer, learn by doing" world, sometimes I wish someone would write an Amibroker AFL Programming for Dummies" book with very detailed code examples (especially for some of the more intricate AFL functions) AND EXPLANATIONS of how the code is structured in simple terminology. The explanations are important .These "Dummies" titled books are very common in the US, and popular for science and math. Amibroker has a large worldwide community, and this might be very useful.
2 Likes
That is because both Osaka plugin and AmiBroker use same (standard) algorithm - QuickSort.
1 Like
@entropy1969 thanks for your nice words.
The @Tomasz answer and the sample code provided by @fxshrat is essentially all you need to replace your Osaka calls with a matrix.
Here, we are very fortunate to have some very skilled programmers that share with us their knowledge and time!
I can not comment on the correctness of the implementation, since I have not examined the original document, but I suggest you carefully evaluate the additional suggestions of @fxshrat because he is usually right.
My code, that as I wrote, was only modified to get an overview of the times spent in different parts of the formula, is essentially a copy of what you posted, with just some extra lines.
For this reason, I think it may not be very useful to other people, so I will send it to you directly.
fxshrat:
I saw Guiseppe's response and thanked him, but in my hurry, had forgotten to say the same to you. I have, like most people, busy days, and am not beyond forgetting sometimes. I know you took quite a bit of time, and I appreciate it. At the moment I responded to Guiseppe, I was thinking about some replies I have seen on trading and other forums, and was referring in a global sense, not characterizing your response....that was the context I intended. We are lucky on this forum, and that's what I meant. I have always been very polite to Tomasz, Marcin, Guiseppe, and anyone else to my knowledge. Please do not assume that I am rude....I have never done that on here. If you believe I have been rude to you on some instance....just communicate with me, and I will try to clarify as I have in this case. That's all that is necessary.
3 Likes
Well, I thought this quote of yours
Sometimes on forums, some people's replies seem to be more about criticism and attitude than they do help.
was referring to other posts in this thread since you did not comment on other posts (which have taken time to create). So I thought you ignored on purpose just because your were given constructive help on code (without any personal criticisms). It was not clear to me that other forums were meant.
Then it was misunderstanding and I have deleted my previous post.
Again no one will make fun of beginners (because being beginner is not bad thing/sin to get punished for. Everyone was beginner some time in past). It is not personal thing to give comment on codes. As I said it is just about cold "boring" technical stuff.
4 Likes
fxshrt:
Thanks for deleting post and understanding. Appreciate input very much
Mike
fxshrt:
Have had more time to read and understand your points on the code. Will need a bit more time on the matrix part, but certainly understand your point on the overkill on those extractions from the table (i.e., the n times loops comment near the end of your first message.
Thank you for the help
Mike
[quote="entropy1969, post:19, topic:6231, full:true"]
fxshrt:
Have had more time to read and understand your points on the code. Will need a bit more time on the matrix part, but certainly understand your point on the overkill on those extractions from the table (i.e., the n times loops comment near the end of your first message.
Thank you for the help
Mike
Hello @beppe Sir,
can u share that code with me.
I am Also search rank with matrix.
Thanks in Advance.
Have a nice time.
@vipul, I prefer not to share it since the original code is not mine.
I only modified it to get an overview of the times spent in different parts of the formula to measure the performance of a matrix implementation vs. Osaka to address this OP remark:
If converting to matrices would be much faster for these purposes, I would like to convert.
This thread explains precisely what to do. Please reread the @fxshrat and @Tomasz posts carefully to learn how to do it yourself: it will be a lot better learning experience than getting the translated formula from me.
On the other hand, if you just would like to sort a matrix keeping also track of a "labels" (strings) column, in the previous post, I shared a way to do it.
In any case, if you need to manage/sort a table with (multiple) text columns, it's probably better to continue using the Osaka plugin.
Thanks sir,
i will try my best.
Thanks.
1 Like
fxshrat, Beppe , Tomasz: (Osaka to Matrix)
Thank all of you for your very substantial help. The code from fxshrat was an enormous benefit (emphasis added) and got me almost to my finish point, as were the comments from Beppe on potential efficiencies from conversion that kept me on this, and of course Tomasz' review of suggestions made in these postings. I had trouble, and put these suggestions away for awhile, but decided I was going to sort this out for the new year since I use several of these types of sorts. Sorted everything out today
Once again, thank all of you very much for your help
Happy New Year,
Michael