Double floating-point precision for all indicator/math calculations

Hello, I am currently using Amibroker to optimize my strategy parameters, and then live-trading with my strategy through a custom python coded bot. I'm running into some minor differences, mainly because Amibroker uses single floating point precision to calculate the indicator values, and the python bot uses double float calculations, so this ends up causing the live-trading signals to be off by ~15-30 minutes from the Amibroker signals.

I'm basically just using all standard TA-lib indicators and normal arithmetic with the outputs to create my trade signals. Is there any possible way to get ALL of the calculations to be in double-floating point precision? I don't care if it takes longer, I just want to compare the strategy backtest metrics between single and double precision and see if there is a big enough difference in profit % etc. to worry about. I know the trades are off by 15-30 minutes but need to see how much discrepancy there is over a year of trades. The strategy averages about 1 trade per day so it's probably going to be a pretty significant difference.

It seems like the only way to do this would be to code a custom plugin for amibroker that would do all the TA-lib indicator calculations and math in double floating point precision, and then feed these values back into amibroker to calculate backtest metrics for the optimizations. Does anyone have any other ideas on how to do this?

Thank you for the help!

Generally there are some misconceptions in your post:

Misconception #1: you are mistaken about internal calculations in AmiBroker.
Single precision is NOT used for calculation of indicator values.
In 99% of cases AmiBroker uses much higher precision for internal calculations including all built-in indicators. What's more AmiBroker internal precision is 80 bit (16-bit more than double precision) in case of 32-bit version because all calculations are done on HARDWARE FPU, that has 80 bit registers. 64-bit compiler on the other hand uses SSE2 and 64-bit precision for internal representation (see link at the end). And yes results coming from 80bit registers may and will be different than coming from 64-bit registers. Such is a nature of floating point standard.

What is single precision used for is the output of function and/or values stored in AFL variables.

Misconception #2. Using double would give you same results as in Python. It won't. Why? Because even Python can give you different values depending on what CPU it is compiled on and with which compiler options and what hardware it is run on (see: https://bugs.python.org/issue2937)
If compiler emits FPU code (x87) then it will use 80 bit internal representation for all floating point numbers (regardless if they were declared as float or double) unless "strict" option is used (slow, and nobody uses it). If compiler emits SSE2 code then it will use 64 bit for doubles and 32 bit for floats but only when there is no automatic promotions (float->double).

Misconception #3. Assumption that TA-lib can be trusted as producing "correct results".
No it can't. TA-lib does not have monopoly on correctness. It is NOT maintained (last update in 2007), version 0.4 (pre-alpha stage?). I looked at it and it is spaghetti code and would never use it for anything.

All in all exact numerical values of ANY indicator (or any floating point calculation for that matter) depend on:

  1. compiler that you use
  2. compiler options
  3. bitness of output and instruction set (x87 vs SSE)
  4. internal algorithm (due to numerical behavior various algorithms may produce different output - TA-lib is known to produce wrong values in some cases)
  5. hardware that you run on (for example floating point run on graphic cards - like NVidia CUDA may produce different output than run say on Intel CPU)

So, all in all you are mistaken in your assumption that what you are asking for will magically make all results from all places equal. If your system is susceptible to 0.0000001 difference then it is a red flag and it should not be traded at all (too risky/too fragile).

More reading: https://stackoverflow.com/questions/3206101/extended-80-bit-double-floating-point-in-x87-not-sse2-we-dont-miss-it

10 Likes

Great thank you for the clarification, I was going by the information available on the amibroker website and it sounded like calculations were done in single-point precision other than "some internal calculations"

In any case, I need the outputs from the indicators to remain in double precision, because I then do multiple operations with them, and that is where my error is multiplying. If I take the raw indicator outputs from my python code and the amibroker code, the results are accurate enough that it doesn't matter, only after multiplying/subtracting/dividing etc do the differences become significant enough to affect trading. This is probably just due to rounding error when converting to single-precision.

Could you add an option to specify the output of the technical indicators remain in double-precision instead of being converted back to single? Also, it would be great if amibroker released a technical indicators library to replace crappy TA-lib. You could charge for it - I'd use it!

Thank you!

The issue is not (only) the speed but RAM usage. You need to keep in mind that you are not the only user. People are backtesting / optimizing large data sets such as millions of bars per symbol and 10000+ symbols. With float, each AFL array is (4 * number of bars) bytes. With double each array is (8 * number of bars) bytes.
With million bars, it means 4MB per array with float vs 8MB per array with double.
This may not seem much but it if you have 10000 static variables (and people use more), you end up with 40 GB vs 80GB. Thats huge difference.
Suddenly system that could be backtested, becomes unbacktestable because of memory usage. So it is not a question of running "slower" but not running at all.

One of possible solutions would be to have two compilations so user could choose whenever he/she is willing to sacrifice 2x memory usage, but maintaining 4 versions
is a extra chore. Also: existing plugins would not be compatible and data still uses floats, so AddToComposite/Foreign would still use float. All in all there are a lot more factors that you are not aware of and as I said, it is very very unlikely that you are really limited by floating point. Do you have university degree in mathematics and have any experience in numerical analysis?

Thank you Tomasz, my question is: is there a possible way to instruct Amibroker to do all calculations in 64 bit, and if not, could this feature be added easily, as a user-selectable option? These are two simple yes/no questions.

And yes I actually do have an advanced university degree in mathematics. Error propagation on rounded off numbers that must be accurate to 6+ decimal places happen very fast when you are adding and subtracting and multiplying and dividing numbers that have been rounded from 64 bit to 32 bit. These are simple concepts.

You answer just determines what step I have to take next in getting my two programs to match exactly, so if you could please answer my question that would be wonderful! :slight_smile:

Thank you very much for your helpful reply.

Show your diploma then because judging from level of your questions I can hardly believe that.
Even last question is absurd for somebody with "university degree in maths".
If you really had it, you would know the answer yourself.
And if you had degree in maths you would know that pretty much all sentences that you wrote show that you don't have degree in maths.
Math practitioner would never write "6+ decimal places" because it is not about decimal places and not 6 digits. You made two fundamental errors in one sentence.

So I am 100% sure that you don't have any clue about what you are talking about and I recommend re-reading articles I pointed out earlier and re-reading previous posts of mine.

As to answer to your questions: everything is possible. Would I do that? Maybe yes maybe not. Certainly NOT for reasons that you presented (because they are not valid).

2 Likes

Here is something for you. Take Microsoft Excel and do the exercise as shown in the video:

  1. In cell A1 enter -1
  2. In cell A2 enter formula =A1+0.1
  3. Copy-paste cell A2 to subsequent rows
  4. You expect to get -0.9, -0.8, -0.7, -0.6, ...., -0.2, -0.1, 0, ...
  5. Look at the surprising value that you get in A11. It should be zero, but it isn't. It is 1.3878E-16

Why? Because double precision arithmetic is NOT the answer. It does not magically solve your "problem". It does not make infinite binary representations of certain fractions suddenly accurate. 0.1 and zillions of other fractions are inaccurate in binary floating point regardless of precision.

video

(Video shows Microsoft Excel in action)

5 Likes

Wow, I've been using Excel for 27 years and never knew that could happen!

Apparently I haven't clarified what I need. I don't CARE if double floating point precision still has some minute errors. I don't NEED the most accurate results possible in the entire universe. What I do NEED is:

The value of my indicator implemented in Amibroker to EXACTLY match the value of the indicator implemented in python, .php, or any other language (they all give the same result), which do the calculation in double-point precision.

So actually, in this situation, double floating point precision IS the answer, because my live trading bot does its calculations in double floating point precision, so that's what I need from my Amibroker model to make the trade times and prices match.

So, once again, how can I accomplish this in Amibroker or do I need to find some other software instead?

Thank you very much!

Re-read what I wrote before because you did not get that at all. Your assumptions are wrong and I explained already why is so and I won't repeat myself anymore.

If you want exactly as in Python call Python:

2 Likes